### Install MLflow

Source: https://github.com/mlflow/skills/blob/main/instrumenting-with-mlflow-tracing/references/python.md

Installs the MLflow library using pip. Ensure you have a version greater than or equal to 3.8.0 for tracing capabilities.

```bash
pip install mlflow>=3.8.0
```

--------------------------------

### Quick MLflow Environment Configuration

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Automatically configures the MLflow environment by running a setup script. This script detects whether to connect to a Databricks or local MLflow server and creates an experiment if one doesn't exist. It outputs environment variables for tracking URI and experiment ID.

```bash
uv run python scripts/setup_mlflow.py
```

--------------------------------

### Install MLflow Tracing

Source: https://github.com/mlflow/skills/blob/main/instrumenting-with-mlflow-tracing/references/typescript.md

Installs the MLflow Tracing npm package. This is the first step to enable tracing in your TypeScript application.

```bash
npm install mlflow-tracing
```

--------------------------------

### Install or Update MLflow

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Verifies the installed MLflow version and installs or updates it to version 3.8.0 or higher if necessary. This ensures compatibility with the required features.

```bash
uv run mlflow --version
```

```bash
uv pip install mlflow>=3.8.0
```

--------------------------------

### Example Diverse Query Set (Python)

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Provides a Python list of dictionaries representing diverse sample queries for an MLflow GenAI dataset. The examples cover variations in complexity, length, topic, and query type to ensure comprehensive testing.

```python
[
    {"inputs": {"query": "What is MLflow?"}},  # Simple, short, basic
    {"inputs": {"query": "How do I log a model?"}},  # Action-oriented
    {
        "inputs": {"query": "What's the difference between experiments and runs?"}
    },  # Comparison
    {
        "inputs": {"query": "Show me an example of using autolog with LangChain"}
    },  # Example request
    {
        "inputs": {
            "query": "How can I track hyperparameters, metrics, and artifacts in a single run?"
        }
    },  # Complex, multi-part
]
```

--------------------------------

### Install and Verify MLflow

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Installs MLflow using uv pip and verifies the installation by checking the MLflow version. Ensure your virtual environment is activated before running these commands.

```bash
uv pip install mlflow
mlflow --version
```

--------------------------------

### Initializing Spark Session for Dataset Creation

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Demonstrates how to initialize a DatabricksSession to enable MLflow-managed dataset creation in Unity Catalog.

```python
from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.remote(serverless=True).getOrCreate()

dataset = mlflow.genai.datasets.create_dataset(
    uc_table_name="catalog.schema.my_dataset"
)
```

--------------------------------

### Set MLflow Tracking URI (Manual)

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Manually sets the MLFLOW_TRACKING_URI environment variable. Examples are provided for Databricks, a local server, and a generic remote server.

```bash
# For Databricks
export MLFLOW_TRACKING_URI="databricks"
```

```bash
# For local server (start server first: mlflow server --host 127.0.0.1 --port 5000 &)
export MLFLOW_TRACKING_URI="http://127.0.0.1:5000"
```

```bash
# For other remote server
export MLFLOW_TRACKING_URI="<your_server_uri>"
```

--------------------------------

### Create New MLflow Experiment

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Creates a new MLflow experiment with a specified name and then retrieves its ID, setting it to the MLFLOW_EXPERIMENT_ID environment variable. The format for the experiment name differs for Databricks compared to local setups.

```bash
# For Databricks - must use /Users/<username>/<name> format
export EXP_NAME="/Users/<your_email>/<experiment_name>"
uv run mlflow experiments create --experiment-name "$EXP_NAME"
```

```bash
# For local - simple name works
export EXP_NAME="<experiment_name>"
uv run mlflow experiments create --experiment-name "$EXP_NAME"
```

```bash
# Get the experiment ID
export EXP_ID=$(uv run python -c "
import mlflow
mlflow.set_tracking_uri('$MLFLOW_TRACKING_URI')
exp = mlflow.get_experiment_by_name('$EXP_NAME')
print(exp.experiment_id)
")
export MLFLOW_EXPERIMENT_ID="$EXP_ID"
```

--------------------------------

### Verify GenAI Traces via CLI

Source: https://github.com/mlflow/skills/blob/main/mlflow-onboarding/SKILL.md

Uses the MLflow CLI to search for and verify that traces are being captured correctly in the experiment store.

```bash
mlflow traces search \
  --experiment-id <EXPERIMENT_ID> \
  --max-results 5 \
  --extract-fields 'info.trace_id,info.state,info.request_time' \
  --output json > /tmp/verify_traces.json
jq '.traces | length' /tmp/verify_traces.json
```

--------------------------------

### Perform Manual Logging and Custom Tracing

Source: https://github.com/mlflow/skills/blob/main/mlflow-onboarding/SKILL.md

Provides methods for logging custom metrics or parameters and instrumenting functions that are not covered by automatic logging.

```python
# Custom tracing for non-supported functions
@mlflow.trace
def my_custom_tool(query: str) -> str:
    # ... tool logic ...
    return result

# Manual logging for metrics/params
with mlflow.start_run():
    mlflow.log_param("custom_param", value)
    mlflow.log_metric("custom_metric", value)
```

--------------------------------

### Manage Local MLflow Server Ports

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Provides solutions for starting the local MLflow server, including checking for existing processes, using a different port, and killing a running server. Also shows how to check port availability.

```bash
ps aux | grep mlflow
mlflow server --port 5051 ...
pkill -f "mlflow server"
lsof -i :5050
```

--------------------------------

### Configure GenAI Autologging

Source: https://github.com/mlflow/skills/blob/main/mlflow-onboarding/SKILL.md

Enables automatic tracing for supported LLM frameworks. This should be called at the application entry point before any LLM interactions occur.

```python
import mlflow

# Pick the one that matches the user's LLM provider:
mlflow.openai.autolog()       # OpenAI SDK
mlflow.anthropic.autolog()    # Anthropic SDK
mlflow.langchain.autolog()    # LangChain / LangGraph
mlflow.litellm.autolog()      # LiteLLM
```

--------------------------------

### Check MLflow Environment Variables

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Checks if MLFLOW_TRACKING_URI and MLFLOW_EXPERIMENT_ID are already set in the current environment. If both are set, it indicates the environment is pre-configured, and subsequent setup steps can be skipped.

```bash
echo "MLFLOW_TRACKING_URI=$MLFLOW_TRACKING_URI"
echo "MLFLOW_EXPERIMENT_ID=$MLFLOW_EXPERIMENT_ID"
```

--------------------------------

### Verify MLflow Configuration and Connection

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Verifies the MLflow environment variables (MLFLOW_TRACKING_URI, MLFLOW_EXPERIMENT_ID) and attempts to connect to the specified MLflow experiment. It prints the status of the connection and experiment details.

```python
import os
import mlflow

tracking_uri = os.getenv('MLFLOW_TRACKING_URI')
experiment_id = os.getenv('MLFLOW_EXPERIMENT_ID')

if tracking_uri and experiment_id:
    print(f'✓ MLFLOW_TRACKING_URI: {tracking_uri}')
    print(f'✓ MLFLOW_EXPERIMENT_ID: {experiment_id}')
    mlflow.set_tracking_uri(tracking_uri)
    try:
        exp = mlflow.get_experiment(experiment_id)
        print(f'✓ Connected to experiment: {exp.name}')
    except Exception as e:
        print(f'⚠ Failed to connect to experiment ID {experiment_id}: {e}')
else:
    print('⚠ Environment variables not set - check agent configuration')
```

--------------------------------

### Configure Traditional ML Autologging

Source: https://github.com/mlflow/skills/blob/main/mlflow-onboarding/SKILL.md

Enables automatic logging of parameters, metrics, and models for standard machine learning frameworks. This should be invoked prior to the training process.

```python
import mlflow

# Pick the one that matches the user's ML framework:
mlflow.sklearn.autolog()      # scikit-learn
mlflow.pytorch.autolog()      # PyTorch / PyTorch Lightning
mlflow.tensorflow.autolog()   # TensorFlow / Keras
mlflow.xgboost.autolog()      # XGBoost
mlflow.lightgbm.autolog()     # LightGBM
```

--------------------------------

### Complete LLM Judge Registration Example

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/scorers-constraints.md

A comprehensive example showing the registration of an LLM judge including name, description, and instructions with proper template variable usage.

```bash
uv run mlflow scorers register-llm-judge \
  -n "RelevanceCheck" \
  -d "Checks if response addresses the query" \
  -i "Given the response {{ outputs }}, determine if it directly addresses the query. Return 'yes' if relevant, 'no' if not."
```

--------------------------------

### Verify MLflow configuration

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Executes a Python script to validate that the MLflow settings are correctly loaded from the configuration module. It asserts the presence of required variables and prints a success message.

```bash
uv run python -c "
from config import settings
assert settings.mlflow_tracking_uri, 'MLFLOW_TRACKING_URI not loaded'
assert settings.mlflow_experiment_id, 'MLFLOW_EXPERIMENT_ID not loaded'
print('✓ MLflow configuration verified')
"
```

--------------------------------

### Using Fully Qualified Names for Unity Catalog Tables

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Ensures dataset creation succeeds by using fully-qualified table names in the format catalog.schema.table.

```python
dataset = create_dataset(name="main.default.my_eval")
```

--------------------------------

### Implement GenAI Evaluation

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Demonstrates the correct usage of the GenAI evaluation API compared to the legacy classic ML evaluation method.

```python
results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=my_app,
    scorers=[Guidelines(name="test", guidelines="...")]
)
```

--------------------------------

### Core MLflow Tracing APIs

Source: https://github.com/mlflow/skills/blob/main/instrumenting-with-mlflow-tracing/references/typescript.md

Demonstrates the core MLflow Tracing APIs including initialization, function wrapping with `mlflow.trace`, creating manual spans with `mlflow.withSpan`, and flushing traces.

```typescript
import * as mlflow from "mlflow-tracing";

// Initialize (required before tracing)
mlflow.init({
    trackingUri: "http://localhost:5000",
    experimentId: "my-experiment",
});

// Function wrapper - creates traced version of a function
const tracedFn = mlflow.trace(
    (arg: string) => { return result; },
    { name: "my_function", spanType: mlflow.SpanType.CHAIN }
);

// Manual span - for dynamic names or non-function code
const result = await mlflow.withSpan(
    { name: "dynamic_span", spanType: mlflow.SpanType.TOOL },
    async (span) => {
        span.setInputs({ key: "value" });
        const result = await doWork();
        span.setOutputs({ result });
        return result;
    }
);

// Flush traces before process exit
await mlflow.flushTraces();
```

--------------------------------

### Authenticate Databricks CLI Profile

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Guides through authenticating a Databricks CLI profile and verifying the authentication status. This is crucial for connecting to Databricks workspaces.

```bash
databricks auth login -p <profile_name>
databricks auth env -p <profile_name>
databricks auth profiles
```

--------------------------------

### Check MLflow CLI Help (CLI)

Source: https://github.com/mlflow/skills/blob/main/retrieving-mlflow-traces/SKILL.md

Displays the help information for the `mlflow traces search` command. It's recommended to run this first to ensure you have the correct flags for your installed MLflow version.

```bash
mlflow traces search --help
```

--------------------------------

### Implement Timeout Handling and Retry Logic

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Provides patterns for handling agent timeouts and rate limits to ensure robust trace collection.

```python
import time

# Timeout handling
try:
    response = run_agent(query, timeout=60)
except TimeoutError:
    print("Query timed out")

# Retry logic
for attempt in range(3):
    try:
        response = run_agent(query)
        break
    except RateLimitError:
        time.sleep(2**attempt)
```

--------------------------------

### Auto-Tracing OpenAI with MLflow

Source: https://github.com/mlflow/skills/blob/main/instrumenting-with-mlflow-tracing/references/typescript.md

Demonstrates how to automatically trace calls to the OpenAI API by wrapping the OpenAI client instance using `mlflow-openai`. This simplifies tracing external service interactions.

```typescript
import { tracedOpenAI } from "mlflow-openai";
import { OpenAI } from "openai";

// Wrap OpenAI client - all calls auto-traced
const openai = tracedOpenAI(new OpenAI());
```

--------------------------------

### Initialize MLflow Guidelines Scorer (Python)

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Shows the correct way to initialize the `Guidelines` scorer, which requires both `name` and `guidelines` parameters. Missing these parameters will result in a `TypeError`.

```python
from mlflow.genai.scorers import Guidelines

scorer = Guidelines(
    name="professional_tone",  # REQUIRED
    guidelines="The response must be professional"  # REQUIRED
)
```

--------------------------------

### Enable MLflow Autologging for Libraries

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Demonstrates the correct placement of MLflow autologging calls, emphasizing that they should occur before library imports. It specifies the correct autologging functions for LangChain and OpenAI.

```python
# CORRECT:
import mlflow

mlflow.langchain.autolog()  # BEFORE library import
from langchain import ChatOpenAI  # library imports after autolog

# WRONG:
from langchain import ChatOpenAI  # library imports before autolog
import mlflow

mlflow.langchain.autolog()  # TOO LATE
```

--------------------------------

### Manage MLflow GenAI Datasets

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Core operations for creating, populating, and utilizing GenAI datasets within MLflow. Includes the required record schema structure.

```python
from mlflow.genai.datasets import create_dataset, search_datasets

# Record schema structure
record = {"inputs": dict, "expectations": dict, "tags": dict}

# Create and populate
dataset = create_dataset(name="my-dataset", experiment_id=["1"])
records = [{"inputs": {"query": "test"}}]
dataset.merge_records(records)

# Search and evaluate
datasets = search_datasets(filter_string="name = 'my-dataset'")
results = mlflow.genai.evaluate(data=datasets[0], predict_fn=predict_fn, scorers=[RelevanceToQuery()])
```

--------------------------------

### Enable MLflow AutoLogging for Frameworks

Source: https://github.com/mlflow/skills/blob/main/instrumenting-with-mlflow-tracing/references/python.md

Enables zero-code instrumentation for supported frameworks like LangChain, OpenAI, and others. This should be called before importing or using the respective libraries.

```python
import mlflow

mlflow.langchain.autolog()  # or openai, anthropic, litellm, etc.
```

```python
import mlflow

# Enable before importing/using the library
mlflow.langchain.autolog()       # LangChain, LangGraph
mlflow.openai.autolog()          # OpenAI SDK
mlflow.anthropic.autolog()       # Anthropic SDK
mlflow.litellm.autolog()         # LiteLLM
mlflow.dspy.autolog()            # DSPy
mlflow.autogen.autolog()         # AutoGen
mlflow.crewai.autolog()          # CrewAI
```

--------------------------------

### Optimize Development with Local Agent Imports

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Replaces slow model serving endpoint calls with direct local module imports to enable faster iteration and debugging.

```python
from plan_execute_agent import AGENT

def predict_fn(messages):
    result = AGENT.predict({"messages": messages})
    if isinstance(result, dict) and "messages" in result:
        for msg in reversed(result["messages"]):
            if msg.get("role") == "assistant":
                return {"response": msg.get("content", "")}
    return {"response": str(result)}
```

--------------------------------

### Implement GenAI Tracing with Mock Data

Source: https://github.com/mlflow/skills/blob/main/mlflow-onboarding/SKILL.md

Demonstrates how to instrument GenAI functions using the @mlflow.trace decorator and manual span management. This approach is ideal for testing tracing logic without requiring external LLM API keys.

```python
import mlflow

mlflow.set_experiment("example-genai-app")

@mlflow.trace
def mock_chat(query: str) -> str:
    with mlflow.start_span(name="retrieve_context") as span:
        context = "Mock retrieved context for: " + query
        span.set_inputs({"query": query})
        span.set_outputs({"context": context})
    with mlflow.start_span(name="generate_response") as span:
        response = "Mock response based on: " + context
        span.set_inputs({"context": context, "query": query})
        span.set_outputs({"response": response})
    return response

mock_chat("What is MLflow?")
```

--------------------------------

### Discover and Create MLflow Datasets (Bash)

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/SKILL.md

These commands are used for preparing evaluation datasets. `list_datasets.py` discovers and recommends existing datasets, while `create_dataset_template.py` generates a script for creating new datasets from test cases. Dataset discovery is mandatory before creating new ones.

```bash
uv run python scripts/list_datasets.py
uv run python scripts/list_datasets.py --format json
uv run python scripts/list_datasets.py --help
uv run python scripts/create_dataset_template.py --test-cases-file test_cases.txt
uv run python scripts/create_dataset_template.py --help
```

--------------------------------

### Search Project for Traditional ML Indicators (Bash)

Source: https://github.com/mlflow/skills/blob/main/mlflow-onboarding/SKILL.md

Searches Python files recursively in the current directory for import statements and usage patterns indicative of traditional ML/deep learning development. It identifies imports from common ML frameworks like scikit-learn, PyTorch, TensorFlow, and MLflow's ML modules, as well as model training method calls.

```bash
grep -rl --include='*.py' -E '(from sklearn|import torch|import tensorflow|import keras|import xgboost|import lightgbm|mlflow\.sklearn|mlflow\.pytorch|mlflow\.tensorflow|\.fit\()' .
```

--------------------------------

### Verify Installed Libraries for MLflow Autologging

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Checks if specific libraries like LangChain are installed using pip. This is a prerequisite for MLflow's autologging functionality to work correctly.

```bash
pip list | grep langchain
```

--------------------------------

### Search Project for GenAI Indicators (Bash)

Source: https://github.com/mlflow/skills/blob/main/mlflow-onboarding/SKILL.md

Searches Python files recursively in the current directory for import statements and usage patterns indicative of GenAI development. It looks for libraries like openai, langchain, and MLflow's GenAI modules, as well as specific function calls related to LLMs.

```bash
grep -rl --include='*.py' -E '(import openai|import anthropic|from langchain|from langgraph|import litellm|from mlflow\.genai|from mlflow\.tracing|mlflow\.openai|mlflow\.langchain|ChatCompletion|chat\.completions)' .
```

--------------------------------

### Correctness Scorer: Providing Expectations in Python

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Illustrates the correct way to use the `Correctness` scorer in MLflow. The WRONG example shows a common mistake where ground truth (expected facts or response) is missing. The CORRECT example demonstrates how to include `expectations` in the `eval_data` for the scorer to function properly.

```python
# Correctness requires expected_facts or expected_response
eval_data = [
    {"inputs": {"query": "What is X?"}}
]
results = mlflow.genai.evaluate(
    data=eval_data,
    predict_fn=my_app,
    scorers=[Correctness()]  # Will fail - no ground truth!
)
```

```python
eval_data = [
    {
        "inputs": {"query": "What is X?"},
        "expectations": {
            "expected_facts": ["X is a platform", "X is open-source"]
        }
    }
]
```

--------------------------------

### Find Existing MLflow Experiment ID

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Searches for an existing MLflow experiment by its name and sets the MLFLOW_EXPERIMENT_ID environment variable. If the experiment is not found, it prints an error message and exits.

```bash
export EXP_NAME="<your_experiment_name>"
export EXP_ID=$(uv run python -c "
import mlflow
mlflow.set_tracking_uri('$MLFLOW_TRACKING_URI')
experiments = mlflow.search_experiments(
    filter_string=\"name = '$EXP_NAME'",
    max_results=1
)
print(experiments[0].experiment_id if experiments else 'NOT_FOUND')
")

if [ "$EXP_ID" = "NOT_FOUND" ]; then
    echo "Experiment not found: $EXP_NAME"
    exit 1
fi

export MLFLOW_EXPERIMENT_ID="$EXP_ID"
```

--------------------------------

### Fetch and Parse MLflow Trace Data

Source: https://github.com/mlflow/skills/blob/main/analyze-mlflow-trace/SKILL.md

Demonstrates how to retrieve a full trace using the MLflow CLI and perform basic analysis on the resulting JSON file using jq.

```bash
# Fetch full trace to a file
mlflow traces get --trace-id <ID> > /tmp/trace.json

# Process the file using jq
jq '.info.state' /tmp/trace.json
jq '.data.spans | length' /tmp/trace.json
```

--------------------------------

### RetrievalGroundedness Scorer: Using RETRIEVER Span Type in Python

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Explains how to correctly configure traces for the `RetrievalGroundedness` scorer. The WRONG example shows an app without the necessary `RETRIEVER` span type. The CORRECT example demonstrates how to use `span_type="RETRIEVER"` in a traced function to ensure the scorer can identify the retrieval step.

```python
# App has no RETRIEVER span type
@mlflow.trace
def my_rag_app(query):
    docs = get_documents(query)  # Not marked as retriever
    return generate_response(docs, query)

# RetrievalGroundedness will fail - can't find retriever spans
```

```python
# Use span_type="RETRIEVER"
@mlflow.trace(span_type="RETRIEVER")
def retrieve_documents(query):
    return [doc1, doc2]

@mlflow.trace
def my_rag_app(query):
    docs = retrieve_documents(query)  # Now has RETRIEVER span
    return generate_response(docs, query)
```

--------------------------------

### Define MLflow settings in Pydantic class

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Adds MLflow configuration fields to a Pydantic Settings class. This allows for type-safe management and validation of tracking URIs and experiment IDs within the application.

```python
# MLflow Configuration
mlflow_tracking_uri: Optional[str] = Field(
	default=None,
	description="MLflow tracking URI (e.g., 'databricks', 'http://localhost:5000')",
)
mlflow_experiment_id: Optional[str] = Field(
	default=None,
	description="MLflow experiment ID for logging traces and evaluation results",
)
```

--------------------------------

### Persist MLflow configuration to .env file

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/setup-guide.md

Appends MLflow tracking URI and experiment ID environment variables to a .env file. This ensures persistent configuration across sessions, though it may override existing environment variables.

```bash
cat >> .env << EOF

# MLflow Configuration
MLFLOW_TRACKING_URI=$MLFLOW_TRACKING_URI
MLFLOW_EXPERIMENT_ID=$MLFLOW_EXPERIMENT_ID
EOF
```

--------------------------------

### Discover Existing Datasets via CLI

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Commands to list and inspect existing datasets using the provided utility script. Supports both table and JSON output formats.

```bash
# List datasets in table format
uv run python scripts/list_datasets.py

# List datasets in machine-readable JSON
uv run python scripts/list_datasets.py --format json
```

--------------------------------

### Configure MLflow Autologging and Tracing

Source: https://context7.com/mlflow/skills/llms.txt

Demonstrates how to initialize MLflow experiments and enable autologging for GenAI providers (OpenAI, Anthropic, LangChain) and traditional ML frameworks (scikit-learn, PyTorch, XGBoost). Includes the use of the @mlflow.trace decorator for custom function instrumentation.

```python
import mlflow

# GenAI application setup
mlflow.set_experiment("my-genai-app")
mlflow.openai.autolog()
mlflow.anthropic.autolog()
mlflow.langchain.autolog()
mlflow.litellm.autolog()

@mlflow.trace
def my_custom_tool(query: str) -> str:
    return result

# Traditional ML setup
mlflow.set_experiment("my-ml-experiment")
mlflow.sklearn.autolog()
mlflow.pytorch.autolog()
mlflow.tensorflow.autolog()
mlflow.xgboost.autolog()
mlflow.lightgbm.autolog()

with mlflow.start_run():
    mlflow.log_param("custom_param", value)
    mlflow.log_metric("custom_metric", value)
```

--------------------------------

### Valid Scorer Aggregations in Python

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Demonstrates the correct usage of aggregation functions for MLflow scorers. It shows which aggregations are valid and provides a Python code example for defining a scorer with these valid options. Invalid aggregations like 'p50' and 'p99' are highlighted as incorrect.

```python
# Only these 6 are valid
@scorer(aggregations=["min", "max", "mean", "median", "variance", "p90"])
def my_scorer(outputs) -> float:
    return 0.5
```

--------------------------------

### Retrieve MLflow Experiment Kind Tag (Bash)

Source: https://github.com/mlflow/skills/blob/main/mlflow-onboarding/SKILL.md

Retrieves the 'mlflow.experimentKind' tag for a given MLflow experiment ID. It uses the 'mlflow experiments get' command to fetch experiment details and 'jq' to parse the JSON output, defaulting to 'not set' if the tag is absent.

```bash
mlflow experiments get --experiment-id <EXPERIMENT_ID> --output json > /tmp/exp_detail.json
jq -r '.tags["mlflow.experimentKind"] // "not set"' /tmp/exp_detail.json
```

--------------------------------

### Fetch Target MLflow Documentation (WebFetch)

Source: https://github.com/mlflow/skills/blob/main/searching-mlflow-docs/SKILL.md

Fetches the specific MLflow documentation file using a path obtained from the index. This step retrieves the content of the documentation, ensuring that only code blocks are returned verbatim. It requires the URL of the `.md` file and a prompt to prevent summarization and extract code accurately.

```tool_code
WebFetch(
  url: "https://mlflow.org/docs/latest/[path].md",
  prompt: "Return all code blocks verbatim. Do not summarize."
)
```

--------------------------------

### Search MLflow Documentation via WebFetch

Source: https://context7.com/mlflow/skills/llms.txt

A workflow for AI agents to retrieve accurate, up-to-date MLflow documentation by first querying the llms.txt index and then fetching specific Markdown files.

```python
# Workflow for searching MLflow docs:
# 1. Fetch llms.txt index to find relevant page paths
WebFetch(
    url="https://mlflow.org/docs/latest/llms.txt",
    prompt="Find links or references to [TOPIC]. List all relevant URLs."
)

# 2. Fetch target documentation (always use .md extension)
WebFetch(
    url="https://mlflow.org/docs/latest/[path].md",
    prompt="Return all code blocks verbatim. Do not summarize."
)
```

--------------------------------

### MLflow Trace Metrics API Response Example (JSON)

Source: https://github.com/mlflow/skills/blob/main/querying-mlflow-metrics/references/api_reference.md

An example JSON response from the MLflow Trace Metrics API, showing aggregated data points. Each data point includes the metric name, dimensions, and aggregated values.

```json
{
  "data_points": [
    {
      "metric_name": "total_tokens",
      "dimensions": {"time_bucket": "2024-01-19T00:00:00+00:00", "trace_name": "chat"},
      "values": {"SUM": 15000, "AVG": 500.0}
    }
  ],
  "next_page_token": null
}
```

--------------------------------

### MLflow Trace Metrics API Request Example (JSON)

Source: https://github.com/mlflow/skills/blob/main/querying-mlflow-metrics/references/api_reference.md

An example JSON payload for requesting trace metrics from the MLflow API. It specifies experiment IDs, view type, metric name, desired aggregations, dimensions for grouping, and time range.

```json
{
  "experiment_ids": ["1"],
  "view_type": 1,
  "metric_name": "total_tokens",
  "aggregations": [
    {"aggregation_type": 2},
    {"aggregation_type": 3}
  ],
  "dimensions": ["trace_name"],
  "time_interval_seconds": 3600,
  "start_time_ms": 1737244800000,
  "end_time_ms": 1737331200000
}
```

--------------------------------

### Create and Manage Evaluation Datasets

Source: https://context7.com/mlflow/skills/llms.txt

Shows how to initialize evaluation datasets, merge input records, and utilize them in MLflow evaluation runs or Databricks Unity Catalog.

```python
from mlflow.genai.datasets import create_dataset, search_datasets
import mlflow

dataset = create_dataset(name="my-agent-eval-v1", experiment_id=["1"])

records = [
    {"inputs": {"query": "What is MLflow?"}},
    {"inputs": {"query": "How do I log a model?"}}
]
dataset.merge_records(records)

results = mlflow.genai.evaluate(
    data=dataset,
    predict_fn=predict_fn,
    scorers=[RelevanceToQuery()]
)
```

--------------------------------

### GET /traces/get

Source: https://github.com/mlflow/skills/blob/main/retrieving-mlflow-traces/SKILL.md

Retrieves a single MLflow trace by its unique identifier.

```APIDOC
## GET /traces/get

### Description
Fetches the full details of a specific trace using its unique trace ID.

### Method
CLI Command: `mlflow traces get`

### Parameters
#### Query Parameters
- **--trace-id** (string) - Required - The unique identifier of the trace to retrieve.

### Request Example
`mlflow traces get --trace-id tr-69f72a3772570019f2f91b75b8b5ded9`

### Response
#### Success Response (200)
- **TraceInfo** (object) - Contains trace_id, status, timestamp_ms, execution_time_ms, tags, metadata, and assessments.
- **Spans** (array) - Tree structure of operations within the trace.
```

--------------------------------

### List Databricks Catalogs (Bash)

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/dataset-preparation.md

Uses the Databricks CLI to list available catalogs in a Databricks environment. This is useful for identifying potential catalog names when configuring datasets for Unity Catalog.

```bash
databricks catalogs list
```

--------------------------------

### Execute MLflow Evaluation Workflow

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/SKILL.md

Commands to generate an evaluation script from a template and execute the resulting evaluation process using the uv package manager.

```bash
# Generate evaluation script (specify module and entry point)
uv run python scripts/run_evaluation_template.py \
  --module mlflow_agent.agent \
  --entry-point run_agent

# Review the generated script, then execute it
uv run python run_agent_evaluation.py
```

--------------------------------

### GET /traces/search

Source: https://github.com/mlflow/skills/blob/main/retrieving-mlflow-traces/SKILL.md

Searches for traces within an experiment based on provided filters, status, or metadata.

```APIDOC
## GET /traces/search

### Description
Searches for multiple traces within an experiment using a filter string for advanced querying.

### Method
CLI Command: `mlflow traces search`

### Parameters
#### Query Parameters
- **--experiment-id** (string) - Required - The ID of the experiment to search.
- **--filter-string** (string) - Optional - SQL-like filter criteria (e.g., "trace.status = 'ERROR'").
- **--max-results** (int) - Optional - Limit the number of results returned.
- **--page-token** (string) - Optional - Token for pagination.
- **--order-by** (string) - Optional - Field to sort results by (e.g., "timestamp_ms ASC").

### Request Example
`mlflow traces search --experiment-id 1 --filter-string "trace.status = 'ERROR'" --max-results 50`

### Response
#### Success Response (200)
- **Traces** (array) - List of matching TraceInfo objects.
- **Next page token** (string) - Token provided if further results exist.
```

--------------------------------

### GET /mlflow/traces/{trace_id}

Source: https://github.com/mlflow/skills/blob/main/analyze-mlflow-trace/references/trace-structure.md

Retrieves the detailed structure of an MLflow trace, including timing information, input/output previews, and associated metadata.

```APIDOC
## GET /mlflow/traces/{trace_id}

### Description
Retrieves the full trace object containing performance metrics, input/output previews, and metadata for a specific trace ID.

### Method
GET

### Endpoint
/mlflow/traces/{trace_id}

### Parameters
#### Path Parameters
- **trace_id** (string) - Required - The unique identifier of the trace.

### Response
#### Success Response (200)
- **request_time** (int) - Timestamp in ms since Unix epoch.
- **execution_duration** (int) - Total wall-clock time in ms.
- **request_preview** (string) - JSON-encoded input preview.
- **response_preview** (string) - JSON-encoded output preview.
- **client_request_id** (string) - Optional external system identifier.
- **trace_location** (object) - Storage location details (MlflowExperimentLocation or UCSchemaLocation).
- **trace_metadata** (dict) - Key-value pairs including mlflow.modelId, mlflow.trace.user, etc.

#### Response Example
{
  "request_time": 1715678400000,
  "execution_duration": 1500,
  "request_preview": "{\"query\": \"What is MLflow?\"}",
  "response_preview": "{\"answer\": \"MLflow is an open source platform...\"}",
  "trace_metadata": {
    "mlflow.modelId": "model-v1",
    "mlflow.trace.user": "user-123"
  }
}
```

--------------------------------

### Fetch MLflow Documentation Index (WebFetch)

Source: https://github.com/mlflow/skills/blob/main/searching-mlflow-docs/SKILL.md

Fetches the `llms.txt` file from the MLflow documentation site to find relevant page paths for LLM/GenAI topics. This step is crucial for identifying the correct documentation to retrieve. It takes a URL as input and uses a prompt to guide the search for specific topics within the index.

```tool_code
WebFetch(
  url: "https://mlflow.org/docs/latest/llms.txt",
  prompt: "Find links or references to [TOPIC]. List all relevant URLs."
)
```

--------------------------------

### Implementing Session Tracking in MLflow Traces

Source: https://github.com/mlflow/skills/blob/main/agent-evaluation/references/troubleshooting.md

Shows how to capture a session_id by updating the current trace within a decorated function.

```python
@mlflow.trace
def run_agent(query: str, session_id: str = None):
    if session_id is None:
        session_id = str(uuid.uuid4())

    # Set tag on the current active trace
    mlflow.update_current_trace(tags={"session_id": session_id})
```

--------------------------------

### Query Trace Metrics and API Reference

Source: https://context7.com/mlflow/skills/llms.txt

Demonstrates how to fetch aggregated metrics like token usage, latency, and quality scores using CLI scripts or direct REST API calls. Useful for monitoring production agent performance.

```bash
python scripts/fetch_metrics.py \
  -s http://localhost:5000 \
  -x 1 \
  -m total_tokens \
  -a SUM,AVG

curl -X POST http://localhost:5000/api/3.0/mlflow/traces/metrics \
  -H "Content-Type: application/json" \
  -d '{
    "experiment_ids": ["1"],
    "metric_name": "total_tokens"
  }'
```