### Run MLflow Setup Script

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Executes the 'setup_mlflow.py' script using 'uv' to automatically detect and configure the MLflow environment. This script handles both Databricks and local MLflow server setups, creating experiments as needed and outputting environment variables.

```bash
uv run python scripts/setup_mlflow.py
```

--------------------------------

### Example Scorer Test Session (Bash)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/scorers.md

This example demonstrates testing the 'ToolUsageAppropriate' scorer with different scenarios. It shows how to test cases where tools should be used, where they shouldn't, and where they were expected but not used, verifying the scorer's output against expected results.

```bash
# Test ToolUsageAppropriate scorer

# Test 1: Query that should use tools (expect: yes)
uv run mlflow traces evaluate \
  --scorers ToolUsageAppropriate \
  --trace-ids <trace_with_tool_use> \
  --output json

# Test 2: Greeting that shouldn't use tools (expect: yes)
uv run mlflow traces evaluate \
  --scorers ToolUsageAppropriate \
  --trace-ids <trace_greeting> \
  --output json

# Test 3: Query that should use tools but didn't (expect: no)
uv run mlflow traces evaluate \
  --scorers ToolUsageAppropriate \
  --trace-ids <trace_missing_tools> \
  --output json
```

--------------------------------

### Configure MLflow Tracking Server and Experiment (Quick Setup)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Automatically detects and configures the MLflow tracking server (Databricks or local) and sets up a project-specific experiment. It exports the MLFLOW_TRACKING_URI and MLFLOW_EXPERIMENT_ID environment variables for subsequent use. This script is recommended for most users.

```bash
# 1. Detect tracking server type
if databricks current-user me &> /dev/null;
    then
    # Databricks detected
    export MLFLOW_TRACKING_URI="databricks"
    export DB_USER=$(databricks current-user me --output json | grep -o '"value":"[^"]*"' | head -1 | cut -d'"' -f4)
    export PROJECT_NAME=$(basename $(pwd))
    export EXP_NAME="/Users/$DB_USER/${PROJECT_NAME}-evaluation"
    echo "✓ Detected Databricks"
    echo "  User: $DB_USER"
    echo "  Experiment: $EXP_NAME"
else
    # Local or other server
    export MLFLOW_TRACKING_URI="http://127.0.0.1:5000"
    export PROJECT_NAME=$(basename $(pwd))
    export EXP_NAME="${PROJECT_NAME}-evaluation"
    echo "✓ Using local MLflow server"
    echo "  URI: $MLFLOW_TRACKING_URI"
    echo "  Experiment: $EXP_NAME"
    echo ""
    echo "  Note: If MLflow server isn\'t running, start it with:"
    echo "    mlflow server --host 127.0.0.1 --port 5000 &"
fi

# 2. Find existing or create new experiment
export EXP_ID=$(uv run python -c "
import mlflow
mlflow.set_tracking_uri('$MLFLOW_TRACKING_URI')
experiments = mlflow.search_experiments(
    filter_string=\"name = '$EXP_NAME'",
    max_results=1
)
if experiments:
    print(experiments[0].experiment_id)
else:
    print(mlflow.create_experiment('$EXP_NAME'))
")

# 3. Display configuration
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "✓ MLflow Configuration Complete"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "Tracking URI:   $MLFLOW_TRACKING_URI"
echo "Experiment ID:  $EXP_ID"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo ""

# Export for use in subsequent steps
export MLFLOW_EXPERIMENT_ID="$EXP_ID"

```

--------------------------------

### Install and Verify MLflow

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Installs MLflow using uv and verifies the installation by checking the MLflow version. Ensures MLflow is accessible in the current environment.

```bash
uv pip install mlflow
mlflow --version
```

--------------------------------

### Create New MLflow Experiment (Bash)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Creates a new MLflow experiment. It provides different formats for experiment names depending on the environment (Databricks vs. local). It then retrieves and exports the newly created experiment's ID. Requires MLflow and uv to be installed.

```bash
# For Databricks - must use /Users/<username>/<name> format
export EXP_NAME="/Users/<your_email>/<experiment_name>"
uv run mlflow experiments create --experiment-name "$EXP_NAME"

# For local - simple name works
export EXP_NAME="<experiment_name>"
uv run mlflow experiments create --experiment-name "$EXP_NAME"

# Get the experiment ID
export EXP_ID=$(uv run python -c "
import mlflow
mlflow.set_tracking_uri('$MLFLOW_TRACKING_URI')
exp = mlflow.get_experiment_by_name('$EXP_NAME')
print(exp.experiment_id)
")
export MLFLOW_EXPERIMENT_ID="$EXP_ID"
```

--------------------------------

### Example Diverse Test Queries

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Provides an example list of diverse input queries for testing MLflow GenAI agents. The queries vary in complexity, length, and type to cover a wide range of agent capabilities and potential edge cases, promoting comprehensive evaluation.

```python
[
    {"inputs": {"query": "What is MLflow?"}},  # Simple, short, basic
    {"inputs": {"query": "How do I log a model?"}},  # Action-oriented
    {
        "inputs": {"query": "What's the difference between experiments and runs?"}
    },  # Comparison
    {
        "inputs": {"query": "Show me an example of using autolog with LangChain"}
    },  # Example request
    {
        "inputs": {
            "query": "How can I track hyperparameters, metrics, and artifacts in a single run?"
        }
    },  # Complex, multi-part
]
```

--------------------------------

### Python: Comprehensive MLflow Tracing Example

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/tracing-integration.md

This Python snippet provides a more detailed example of MLflow tracing integration. It includes enabling autologging before imports, decorating the entry point function, and optionally tracking sessions for multi-turn conversations.

```python
# step 1: Enable autolog BEFORE imports
import mlflow
mlflow.langchain.autolog()  # Or langgraph, openai, etc. Use the documentation protocol to find the integration for different libraries.

# step 2: Import agent code
from my_agent import agent

# step 3: Add @mlflow.trace decorator
@mlflow.trace
def run_agent(query: str, session_id: str = None) -> str:
    """Agent entry point with tracing."""
    result = agent.run(query)

    # step 4 (optional): Track session for multi-turn
    if session_id:
        trace_id = mlflow.get_last_active_trace_id()
        if trace_id:
            mlflow.set_trace_tag(trace_id, "session_id", session_id)

    return result
```

--------------------------------

### Install and Configure MLflow Tracing (TypeScript)

Source: https://github.com/b-step62/skills/blob/main/instrumenting-with-mlflow-tracing/SKILL.md

Installs the MLflow Tracing npm package and initializes the tracing configuration with the tracking URI and experiment ID. This is the TypeScript equivalent for setting up MLflow tracing.

```bash
npm install mlflow-tracing
```

```typescript
import * as mlflow from "mlflow-tracing";

mlflow.init({
    trackingUri: "http://localhost:5000",
    experimentId: "my-agent",
});
```

--------------------------------

### Check Installed Python Packages

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Lists installed Python packages and filters for specific libraries like 'langchain'. Useful for verifying that required libraries are installed and accessible.

```bash
pip list | grep langchain
```

--------------------------------

### MLflow Tracing Setup (Bash)

Source: https://context7.com/b-step62/skills/llms.txt

Installs MLflow and configures the tracking URI and experiment name for local or Databricks deployments. Ensures the MLflow client is connected to the correct server and experiment.

```bash
# Install MLflow
pip install mlflow>=3.8.0

# For Databricks
export MLFLOW_TRACKING_URI="databricks"

# For local server
export MLFLOW_TRACKING_URI="http://127.0.0.1:5000"

# Set experiment
export MLFLOW_EXPERIMENT_NAME="my-agent"
```

--------------------------------

### Python: Basic MLflow Tracing Setup

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/tracing-integration.md

This snippet demonstrates the minimum implementation for MLflow tracing in Python. It shows how to enable autologging before importing agent code and how to decorate the agent's main function with `@mlflow.trace`.

```python
import mlflow
mlflow.langchain.autolog()  # Before imports

from my_agent import agent

@mlflow.trace
def run_agent(query: str) -> str:
    return agent.run(query)
```

--------------------------------

### Verify Python Package Installation

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Lists installed Python packages and filters for a specific package name to verify its installation. This command is useful for confirming that an agent package has been successfully installed in the current environment.

```bash
pip list | grep <package_name>
```

--------------------------------

### Install or Upgrade MLflow Package

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Installs or upgrades the MLflow Python package to version 3.8.0 or higher using the 'uv' package manager. This ensures compatibility with the required features for agent evaluation.

```bash
uv pip install mlflow>=3.8.0
```

--------------------------------

### Check Existing MLflow Datasets via CLI

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Provides command-line instructions to list and inspect existing MLflow evaluation datasets. It shows how to get table-formatted output and machine-readable JSON output.

```bash
uv run python scripts/list_datasets.py  # Table format (default)
# Or for machine-readable output:
uv run python scripts/list_datasets.py --format json
```

--------------------------------

### Install and Configure MLflow Tracing (Python)

Source: https://github.com/b-step62/skills/blob/main/instrumenting-with-mlflow-tracing/SKILL.md

Installs the MLflow Python package and sets up the tracking URI and experiment for tracing. Requires MLflow version 3.8.0 or higher.

```bash
pip install mlflow>=3.8.0
```

```python
import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("my-agent")
```

--------------------------------

### Example LLM Judge Instructions

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/scorers.md

Provides example instructions for an LLM judge to evaluate agent responses based on query, response, and trace. It specifies the criteria for evaluation and the expected 'yes'/'no' output format.

```text
You are evaluating whether an agent used appropriate tools for a query.

Given:
- Query: {query}
- Response: {response}
- Trace: {trace} (contains tool calls)

Criteria: The agent should use tools when needed (e.g., search for factual queries)
and should not use tools unnecessarily (e.g., for greetings).

Evaluate whether appropriate tools were used. Return "yes" if tools were used
appropriately, "no" if not.
```

--------------------------------

### Find Existing MLflow Experiment by Name (Bash)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Searches for an existing MLflow experiment by its name and exports its ID. It handles cases where the experiment is not found. Requires MLflow to be installed and MLFLOW_TRACKING_URI to be set.

```bash
export EXP_NAME="<your_experiment_name>"
export EXP_ID=$(uv run python -c "
import mlflow
mlflow.set_tracking_uri('$MLFLOW_TRACKING_URI')
experiments = mlflow.search_experiments(
    filter_string=\"name = '$EXP_NAME'\",
    max_results=1
)
print(experiments[0].experiment_id if experiments else 'NOT_FOUND')
")

if [ "$EXP_ID" = "NOT_FOUND" ]; then
    echo "Experiment not found: $EXP_NAME"
    exit 1
fi

export MLFLOW_EXPERIMENT_ID="$EXP_ID"
```

--------------------------------

### Search MLflow Traces using CLI

Source: https://context7.com/b-step62/skills/llms.txt

This section provides examples of how to search and filter MLflow traces using the command-line interface (CLI). It covers searching by status, execution time, tags, metadata, time range, and performing full-text searches. The output can be formatted as JSON. Ensure the MLflow CLI is installed and configured.

```bash
# Search by status
mlflow traces search --experiment-id 1 --filter-string "trace.status = 'ERROR'"

# Search slow traces (> 1 second)
mlflow traces search --experiment-id 1 --filter-string "trace.execution_time_ms > 1000"

# Search by tag
mlflow traces search --experiment-id 1 --filter-string "tag.environment = 'production'"

# Search by metadata (escape special chars with backticks)
mlflow traces search --experiment-id 1 --filter-string "metadata.`user.id` = 'abc'"

# Time range filter (last hour)
mlflow traces search --experiment-id 1 \
  --filter-string "trace.timestamp_ms > $(( $(date +%s)000 - 3600000 ))"

# Combine conditions
mlflow traces search --experiment-id 1 \
  --filter-string "trace.status = 'ERROR' AND trace.execution_time_ms > 500" \
  --output json

# Full text search
mlflow traces search --experiment-id 1 --filter-string "trace.text LIKE '%error%'"

# Get single trace details
mlflow traces get --trace-id <trace_id>
```

--------------------------------

### Discover Built-in Scorers via Documentation

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/scorers.md

This method demonstrates how to query MLflow documentation to discover available built-in scorers. It involves using a WebFetch command with a specific prompt to retrieve information from the MLflow documentation, guiding users to understand scorer names, import paths, evaluation criteria, and required inputs.

```text
WebFetch https://mlflow.org/docs/latest/llms.txt with prompt:
"What built-in LLM judges or scorers are available in MLflow for evaluating GenAI agents?"
```

--------------------------------

### Verify MLflow Configuration

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Verifies the MLflow tracking URI and experiment ID by checking environment variables and attempting to connect to the MLflow experiment. This script is useful for confirming that the automatic configuration has been applied correctly.

```bash
uv run python -c "
import os
import mlflow

tracking_uri = os.getenv('MLFLOW_TRACKING_URI')
experiment_id = os.getenv('MLFLOW_EXPERIMENT_ID')

if tracking_uri and experiment_id:
    print(f'✓ MLFLOW_TRACKING_URI: {tracking_uri}')
    print(f'✓ MLFLOW_EXPERIMENT_ID: {experiment_id}')
    mlflow.set_tracking_uri(tracking_uri)
    exp = mlflow.get_experiment(experiment_id)
    print(f'✓ Connected to experiment: {exp.name}')
else:
    print('⚠ Environment variables not set - check agent configuration')
"
```

--------------------------------

### MLflow Judge Registration Example (Bash)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/scorers-constraints.md

A comprehensive example of registering an MLflow LLM judge that adheres to all constraints. It includes a name, description, and instructions with template variables and 'yes'/'no' return values, demonstrating best practices for CLI usage.

```bash
uv run mlflow scorers register-llm-judge \
  -n "RelevanceCheck" \
  -d "Checks if response addresses the query" \
  -i "Given the response {{ outputs }}, determine if it directly addresses the query. Return 'yes' if relevant, 'no' if not."
```

```bash
uv run mlflow scorers register-llm-judge \
  -n "ToolUsageCheck" \
  -d "Evaluates tool selection quality" \
  -i "Examine the trace {{ trace }}. Did the agent use appropriate tools for the query? Return 'yes' if appropriate, 'no' if not."
```

--------------------------------

### Verify MLflow Configuration Loading (Python)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Verifies that the MLflow tracking URI and experiment ID have been successfully loaded into the application settings. It uses assertions to check for the presence of these values and prints a success message if they are loaded. Requires the 'config' module and Pydantic settings to be set up.

```python
uv run python -c "
from config import settings
assert settings.mlflow_tracking_uri, 'MLFLOW_TRACKING_URI not loaded'
assert settings.mlflow_experiment_id, 'MLFLOW_EXPERIMENT_ID not loaded'
print('✓ MLflow configuration verified')
"
```

--------------------------------

### MLflow Tracing Setup (Python)

Source: https://context7.com/b-step62/skills/llms.txt

Configures MLflow tracking URI and experiment programmatically in Python. Verifies the MLflow version and retrieves the experiment ID for confirmation.

```python
import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("my-agent")

# Verify setup
print(f"MLflow {mlflow.__version__} installed")
exp = mlflow.get_experiment_by_name("my-agent")
print(f"Experiment ID: {exp.experiment_id}")
```

--------------------------------

### Filter MLflow Traces by Time Range

Source: https://github.com/b-step62/skills/blob/main/searching-mlflow-traces/SKILL.md

Searches MLflow traces that occurred within the last hour, using a dynamic timestamp filter. The example shows how to calculate the start time in milliseconds since the epoch.

```bash
# Time range filter (timestamps in milliseconds since epoch)
# Get current time in ms: $(date +%s)000
# Last hour: $(( $(date +%s)000 - 3600000 ))
mlflow traces search --experiment-id 1 --filter-string "trace.timestamp_ms > $(( $(date +%s)000 - 3600000 ))"
```

--------------------------------

### Combine Autologging with Custom Tracing in Python

Source: https://github.com/b-step62/skills/blob/main/instrumenting-with-mlflow-tracing/SKILL.md

This Python example illustrates how to integrate MLflow's autologging for libraries like Langchain with custom trace instrumentation. It uses `mlflow.langchain.autolog()` to automatically log Langchain components and `@mlflow.trace` for custom pipeline steps.

```python
import mlflow
from mlflow.entities import SpanType
from langchain_openai import ChatOpenAI

mlflow.langchain.autolog()

@mlflow.trace(name="rag_pipeline", span_type=SpanType.CHAIN)
def rag_query(question: str) -> str:
    docs = retrieve_documents(question)  # Custom function

    llm = ChatOpenAI()  # Auto-traced by autolog
    response = llm.invoke(format_prompt(docs, question))

    return response.content
```

--------------------------------

### Manage Local MLflow Server

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Provides commands to check for running MLflow server processes, start a server on a different port, and kill an existing server process. Useful for resolving port conflicts.

```bash
ps aux | grep mlflow
mlflow server --port 5051 ...
pkill -f "mlflow server"
lsof -i :5050
```

--------------------------------

### MLflow Filter Syntax Example (Plain Text)

Source: https://github.com/b-step62/skills/blob/main/querying-mlflow-metrics/references/api_reference.md

Examples demonstrating the filter syntax used with the MLflow Trace Metrics API. These filters allow for precise selection of data based on trace status, tags, metadata, span properties, and assessment details.

```text
trace.status = "OK"
trace.tag.<key> = "<value>"
trace.metadata.<key> = "<value>"
span.name = "<value>"
span.type = "LLM"
assessment.name = "<value>"
```

--------------------------------

### Generate Dataset Template Script

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Executes a Python script to generate a dataset template. It can optionally configure the dataset for Databricks Unity Catalog by specifying catalog, schema, and table names. The script guides users through naming conventions and creates diverse sample queries.

```bash
uv run python scripts/create_dataset_template.py --test-cases-file test_cases.txt

# For Databricks Unity Catalog:
uv run python scripts/create_dataset_template.py \
  --test-cases-file test_cases.txt \
  --catalog main --schema ml --table eval_v1
```

--------------------------------

### Verify MLflow Autologging and Tracing Decorators

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Searches for autologging calls and @mlflow.trace decorators within the source code. Helps confirm that tracing is configured correctly in the project.

```bash
grep -r "autolog" src/
grep -r "@mlflow.trace" src/
```

--------------------------------

### Search for MLflow Decorators

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Searches the source code for function definitions and the preceding lines, specifically looking for MLflow decorators. This helps in verifying the correct placement and usage of `@mlflow.trace`.

```bash
grep -B 2 "def run_agent" src/*/agent/*.py
```

--------------------------------

### Validate MLflow Installation and Version

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/SKILL.md

Provides commands to verify the MLflow installation and check its version. This is a pre-flight validation step to ensure that MLflow is installed correctly and meets the minimum version requirement (3.8.0) before proceeding with agent evaluation tasks.

```bash
uv run mlflow --version  # Should be >=3.8.0
uv run python -c "import mlflow; print(f'MLflow {mlflow.__version__} installed')"
```

--------------------------------

### MLflow Manual Spans (TypeScript)

Source: https://github.com/b-step62/skills/blob/main/instrumenting-with-mlflow-tracing/SKILL.md

Illustrates manual span creation in TypeScript using `mlflow.withSpan` for situations where decorators cannot be applied. This method requires explicitly setting inputs and outputs for the span.

```typescript
const result = await mlflow.withSpan(
    { name: `process_${itemId}` },
    async (span) => {
        span.setInputs({ query });
        const result = await process(query);
        span.setOutputs({ result });
        return result;
    }
);
```

--------------------------------

### Configure MLflow Autologging for LangChain

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Demonstrates the correct order for importing MLflow and enabling autologging before importing libraries like LangChain. Incorrect ordering can prevent autologging from capturing spans.

```python
# CORRECT:
import mlflow

mlflow.langchain.autolog()  # BEFORE library import
from langchain import ChatOpenAI  # library imports after autolog

# WRONG:
from langchain import ChatOpenAI  # library imports before autolog
import mlflow

mlflow.langchain.autolog()  # TOO LATE
```

--------------------------------

### Bash: Finding Trace Decorators in Project

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/tracing-integration.md

This command recursively searches the current directory for lines containing `@mlflow.trace`, used to confirm that agent entry points are correctly decorated for tracing.

```bash
# Find trace decorators
grep -r "@mlflow.trace" .
```

--------------------------------

### List Databricks Schemas in a Catalog

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Command to list all schemas within a specified catalog in Databricks. This helps in verifying the existence of schemas when troubleshooting Unity Catalog table not found errors.

```bash
databricks schemas list <catalog>
```

--------------------------------

### Check MLflow CLI Usage

Source: https://github.com/b-step62/skills/blob/main/searching-mlflow-traces/SKILL.md

Displays help information for the 'mlflow traces search' command. This is essential to understand the available flags and options for the installed MLflow version before executing search queries.

```bash
mlflow traces search --help
```

--------------------------------

### Understand Project Structure with File System Commands

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/SKILL.md

Provides bash commands to help understand the overall project structure. This includes checking package configuration files for entry points, reading README and documentation files, and listing key directories.

```bash
# Check entry points in package config
cat pyproject.toml setup.py 2>/dev/null | grep -A 5 "scripts\|entry_points"

# Read project documentation
cat README.md docs/*.md 2>/dev/null | head -100

# Explore main directories
ls -la src/ app/ agent/ 2>/dev/null
```

--------------------------------

### Discover Agent Entry Points and Tracing Integration

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/SKILL.md

Offers bash commands to dynamically explore a project's structure, identify potential agent entry points (main functions), and locate MLflow tracing integrations. This helps in understanding how the agent is structured and how tracing is implemented.

```bash
# Search for main agent functions
grep -r "def.*agent" . --include="*.py"
grep -r "def (run|stream|handle|process)" . --include="*.py"

# Check common locations
ls main.py app.py src/*/agent.py 2>/dev/null

# Look for API routes
grep -r "@app\.(get|post)" . --include="*.py"  # FastAPI/Flask
grep -r "def.*route" . --include="*.py"

# Find autolog calls
grep -r "mlflow.*autolog" . --include="*.py"

# Find trace decorators
grep -r "@mlflow.trace" . --include="*.py"

# Check imports
grep -r "import mlflow" . --include="*.py"
```

--------------------------------

### Run MLflow and Python Commands with uv

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/SKILL.md

Demonstrates the correct way to execute MLflow CLI commands and Python scripts using the 'uv run' command. This ensures that commands are executed within the proper environment with all necessary dependencies, crucial for consistent and reproducible results.

```bash
uv run mlflow --version          # MLflow CLI commands
uv run python scripts/xxx.py     # Python script execution
uv run python -c "..."           # Python one-liners
```

--------------------------------

### Install Agent Package in Editable Mode

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Installs a Python package in editable mode, allowing for immediate use of the package during development without reinstallation after code changes. This command is typically run from the project's root directory.

```bash
pip install -e .
```

--------------------------------

### Programmatic Scorer Registration with make_judge()

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/scorers.md

Illustrates how to programmatically register an LLM judge scorer using the `make_judge` function in Python. This example defines a 'ToolUsageAppropriate' scorer with specific instructions and registers it for an experiment.

```python
from mlflow.genai.judges import make_judge
from typing import Literal

scorer = make_judge(
    name="ToolUsageAppropriate",
    description="Judges whether appropriate tools were used",
    instructions="""
    You are evaluating tool usage by an AI agent.

    Given a trace: {{ trace }}

    Determine if appropriate tools were used.
    Return "yes" if tool usage was appropriate, "no" if not.
    """,
    feedback_value_type=Literal["yes", "no"],
)

# Register the scorer
registered_scorer = scorer.register(experiment_id="your_experiment_id")
```

--------------------------------

### MLflow Manual Spans (Python)

Source: https://github.com/b-step62/skills/blob/main/instrumenting-with-mlflow-tracing/SKILL.md

Demonstrates manual span creation in Python using `mlflow.start_span` for scenarios where decorators are not feasible, such as script-level code or dynamic span naming. Requires manual input and output setting.

```python
with mlflow.start_span(name=f"process_{item_id}") as span:
    span.set_inputs({"query": query})  # Must set manually
    result = process(query)
    span.set_outputs({"result": result})  # Must set manually
```

--------------------------------

### Examine Agent Function Signature and Prepare Dataset Inputs

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/dataset-preparation.md

This snippet demonstrates how to analyze an agent's function signature to correctly map parameters to the 'inputs' dictionary for an MLflow evaluation dataset. It highlights which parameters should come from the dataset and which are provided by the agent's code.

```python
# Load config
config = load_config()

# Agent code
def run_agent(
    query: str, llm_provider: LLMProvider, session_id: str | None = None
) -> str:
    ...

run_agent(query="what can you help me with", llm_provider=config.provider)

# Correct dataset input format:
{"inputs": {"query": "your question here"}}  # ✅ CORRECT - matches parameter name

# Incorrect dataset input formats:
# {"inputs": {"request": "..."}}  # ❌ WRONG - no 'request' parameter
# {"inputs": {"question": "..."}}  # ❌ WRONG - no 'question' parameter
# {"inputs": {"prompt": "..."}}  # ❌ WRONG - no 'prompt' parameter
```

--------------------------------

### Get Last Active Trace ID in Python

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/tracing-integration.md

Retrieves the trace ID of the last active trace within the current MLflow context. This is useful for associating subsequent actions or tags with a specific trace.

```python
trace_id = mlflow.get_last_active_trace_id()
```

--------------------------------

### Set MLflow Tracking URI (Manual)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Manually sets the MLflow tracking URI. This is used when the automatic detection fails or when connecting to a custom remote MLflow server, a non-standard local server, or a Databricks environment.

```bash
# For Databricks
export MLFLOW_TRACKING_URI="databricks"

# For local server (start server first: mlflow server --host 127.0.0.1 --port 5000 &)
export MLFLOW_TRACKING_URI="http://127.0.0.1:5000"

# For other remote server
export MLFLOW_TRACKING_URI="<your_server_uri>"
```

--------------------------------

### Fetch Target MLflow Documentation (WebFetch)

Source: https://github.com/b-step62/skills/blob/main/searching-mlflow-docs/SKILL.md

Fetches the specific MLflow documentation file using a path obtained from the llms.txt index. The file is expected to have a .md extension. This step uses the WebFetch tool with a dynamically constructed URL and a prompt to ensure all code blocks are returned verbatim without summarization.

```tool_code
WebFetch(
  url: "https://mlflow.org/docs/latest/[path].md",
  prompt: "Return all code blocks verbatim. Do not summarize."
)
```

--------------------------------

### Persist MLflow Configuration to .env File (Bash)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Appends MLflow tracking URI and experiment ID to the .env file. This ensures that these critical configuration parameters are saved for future use. Requires the .env file to exist or be created.

```bash
cat >> .env << EOF

# MLflow Configuration
MLFLOW_TRACKING_URI=$MLFLOW_TRACKING_URI
MLFLOW_EXPERIMENT_ID=$MLFLOW_EXPERIMENT_ID
EOF
```

--------------------------------

### Example Custom Scorer Definition - Factual Accuracy

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/scorers.md

Defines a custom scorer named 'FactualAccuracy' to evaluate the factual correctness of an AI agent's response based on the provided context. It specifies the required variables and the 'yes'/'no' output format.

```text
Scorer Name: FactualAccuracy

Definition: Judges whether the response is factually accurate.

Instructions:
You are evaluating the factual accuracy of an AI agent's response.

Review the agent's response and determine if the information provided is
factually correct based on the context and your knowledge.

Return "yes" if the response is factually accurate, "no" if it contains
incorrect information or makes unsupported claims.

Variables: query, response, context (optional)
Output: yes/no
```

--------------------------------

### Create New MLflow Dataset Template via CLI

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Illustrates the command to create a new MLflow evaluation dataset template using a specified test cases file. This is part of the workflow when no suitable existing dataset is found.

```bash
python scripts/create_dataset_template.py --test-cases-file test_cases.txt
```

--------------------------------

### Fetch MLflow Documentation Index (WebFetch)

Source: https://github.com/b-step62/skills/blob/main/searching-mlflow-docs/SKILL.md

Fetches the llms.txt file from the MLflow documentation site to find relevant page paths for LLM/GenAI topics. This step is crucial for identifying the correct documentation to retrieve in the subsequent step. It uses the WebFetch tool with a specific URL and a prompt to guide the search for relevant links.

```tool_code
WebFetch(
  url: "https://mlflow.org/docs/latest/llms.txt",
  prompt: "Find links or references to [TOPIC]. List all relevant URLs."
)
```

--------------------------------

### Define MLflow Configuration in Python Settings (Python)

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/setup-guide.md

Defines Pydantic fields for MLflow tracking URI and experiment ID within a Python Settings class. These fields are optional and include descriptions for clarity. This is typically used in conjunction with a configuration management library.

```python
# MLflow Configuration
mlflow_tracking_uri: Optional[str] = Field(
    default=None,
    description="MLflow tracking URI (e.g., 'databricks', 'http://localhost:5000')",
)
mlflow_experiment_id: Optional[str] = Field(
    default=None,
    description="MLflow experiment ID for logging traces and evaluation results",
)
```

--------------------------------

### Create MLflow GenAI Evaluation Datasets

Source: https://context7.com/b-step62/skills/llms.txt

This Python code demonstrates how to create datasets for systematic agent evaluation using the MLflow GenAI datasets API. It includes searching for existing datasets and creating a new one with specified parameters. The `create_dataset` function requires a name and experiment ID, and the `inputs` dictionary keys must match the agent function's parameter names. Dependencies include `mlflow.genai.datasets` and `os`.

```python
from mlflow.genai.datasets import create_dataset, search_datasets
import os

# Check for existing datasets first
datasets = search_datasets(filter_string="name LIKE '%eval%'')
for ds in datasets:
    print(f"Found: {ds.name} ({len(ds.to_df())} records)")

# Create new dataset with test cases
# IMPORTANT: inputs dict keys MUST match your agent function parameter names
dataset = create_dataset(
    name="my-agent-eval-v1",
    experiment_id=os.getenv("MLFLOW_EXPERIMENT_ID")
)

```

--------------------------------

### Example Custom Scorer Definition - Tool Usage Appropriateness

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/scorers.md

Defines a custom scorer named 'ToolUsageAppropriate' that judges whether an AI agent used appropriate tools for a given query, considering the trace. It specifies the required variables and the 'yes'/'no' output format.

```text
Scorer Name: ToolUsageAppropriate

Definition: Judges whether the agent used appropriate tools for the query.

Instructions:
You are evaluating tool usage by an AI agent.

Given a query and trace showing tool calls, determine if:
1. Tools were used when needed (factual questions, searches, lookups)
2. Tools were NOT used unnecessarily (greetings, simple questions)
3. The RIGHT tools were chosen for the task

Return "yes" if tool usage was appropriate, "no" if not.

Variables: query, response, trace
Output: yes/no
```

--------------------------------

### Core MLflow GenAI Dataset Operations

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Demonstrates essential MLflow GenAI dataset operations including creating a dataset, adding records, searching for existing datasets, and using a dataset for evaluation with a prediction function and scorers.

```python
from mlflow.genai.datasets import create_dataset, search_datasets
import mlflow

# CREATE dataset
dataset = create_dataset(name="my-dataset", experiment_id=["1"])

# ADD records
records = [{"inputs": {"query": "test"}}]
dataset.merge_records(records)

# LOAD dataset
datasets = search_datasets(filter_string="name = 'my-dataset'")
dataset = datasets[0]

# USE in evaluation
# Assuming predict_fn and RelevanceToQuery are defined elsewhere
# results = mlflow.genai.evaluate(
#     data=dataset, predict_fn=predict_fn, scorers=[RelevanceToQuery()]
# )
```

--------------------------------

### Databricks Unity Catalog Dataset Creation

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Demonstrates how to create a dataset specifically for Databricks Unity Catalog. It emphasizes using the fully-qualified table name and omitting the 'tags' parameter, as tags are managed by Unity Catalog itself. This pattern ensures compatibility with Databricks environments.

```python
# Use fully-qualified UC table name, no tags
dataset = create_dataset(
    name="main.default.mlflow_agent_eval_v1",
    experiment_id="<experiment_id>",
    # Note: No tags parameter
)
```

--------------------------------

### Prepare Evaluation Dataset Scripts

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/SKILL.md

These bash commands are used to discover existing evaluation datasets or generate a template for creating a new one. The `list_datasets.py` script helps avoid duplication, while `create_dataset_template.py` assists in generating a new dataset script using MLflow APIs.

```bash
uv run python scripts/list_datasets.py  # Lists, compares, recommends datasets
uv run python scripts/list_datasets.py --format json  # Machine-readable output
uv run python scripts/list_datasets.py --help  # All options

# Generates dataset creation script from test cases file
uv run python scripts/create_dataset_template.py --test-cases-file test_cases.txt
uv run python scripts/create_dataset_template.py --help  # See all options
```

--------------------------------

### MLflow Trace Metrics API Response Example (JSON)

Source: https://github.com/b-step62/skills/blob/main/querying-mlflow-metrics/references/api_reference.md

An example JSON response from the MLflow Trace Metrics API, illustrating the structure of returned data points. Each data point includes the metric name, dimensions, and aggregated values.

```json
{
  "data_points": [
    {
      "metric_name": "total_tokens",
      "dimensions": {"time_bucket": "2024-01-19T00:00:00+00:00", "trace_name": "chat"},
      "values": {"SUM": 15000, "AVG": 500.0}
    }
  ],
  "next_page_token": null
}
```

--------------------------------

### Bash: Querying MLflow Tracing Documentation

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/tracing-integration.md

This command uses `curl` to fetch the latest MLflow documentation for LLMs and then uses `grep` to find information related to tracing. This is a mandatory step before implementing tracing.

```bash
# Query llms.txt for tracing documentation
curl https://mlflow.org/docs/latest/llms.txt | grep -A 20 "tracing"
```

--------------------------------

### MLflow Trace Metrics API Request Example (JSON)

Source: https://github.com/b-step62/skills/blob/main/querying-mlflow-metrics/references/api_reference.md

An example JSON payload for requesting trace metrics from the MLflow API. It specifies experiment IDs, view type, metric name, desired aggregations, dimensions for grouping, and time interval for analysis.

```json
{
  "experiment_ids": ["1"],
  "view_type": 1,
  "metric_name": "total_tokens",
  "aggregations": [
    {"aggregation_type": 2},
    {"aggregation_type": 3}
  ],
  "dimensions": ["trace_name"],
  "time_interval_seconds": 3600,
  "start_time_ms": 1737244800000,
  "end_time_ms": 1737331200000
}
```

--------------------------------

### Set Trace Tag in Python

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/tracing-integration.md

Adds a key-value tag to a specified trace. This function requires the trace ID, the tag key, and the tag value as parameters.

```python
mlflow.set_trace_tag(trace_id, key, value)
```

--------------------------------

### Authenticate Databricks Profile

Source: https://github.com/b-step62/skills/blob/main/agent-evaluation/references/troubleshooting.md

Logs into Databricks CLI with a specified profile and verifies the authentication by displaying the environment variables. This is crucial for connecting to Databricks for MLflow tracking.

```bash
databricks auth login -p <profile_name>
databricks auth env -p <profile_name>
databricks auth profiles
```