### Install MLflow Helm Chart with Example Configuration

Source: https://mlflow.org/docs/latest/self-hosting/kubernetes-helm

Install the MLflow Helm chart using a production-oriented example configuration file.

```bash
helm install mlflow ./charts \
  --namespace mlflow \
  --create-namespace \
  -f charts/example-mlflow-charts.yaml
```

--------------------------------

### Install and Start MLflow

Source: https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/custom-judges/create-custom-judge

Install the MLflow library and start the MLflow server. This is a prerequisite for using the MLflow UI and SDK.

```bash
pip install mlflow  

mlflow server  
```

--------------------------------

### Install MLflow Helm Chart (Quick Start)

Source: https://mlflow.org/docs/latest/self-hosting/kubernetes-helm

Installs the MLflow Helm chart with SQLite backend and local artifact storage for a quick setup. Ensure the chart is downloaded to your current directory.

```bash
helm install mlflow ./charts \
  --namespace mlflow \
  --create-namespace \
  --set storage.enabled=true \
  --set mlflow.backendStoreUri="sqlite:////mlflow/mlflow.db" \
  --set mlflow.artifactsDestination="/mlflow/artifacts"
```

--------------------------------

### Install MLflow and Start Server (pip)

Source: https://mlflow.org/docs/latest/genai/eval-monitor/running-evaluation/traces

Install the MLflow Python package using pip and start a local MLflow server. This is an alternative to using uvx.

```shell
pip install --upgrade mlflow  

mlflow server  
```

--------------------------------

### Install Agno and Dependencies

Source: https://mlflow.org/docs/latest/genai/tracing/integrations/listing/agno

Install the necessary libraries for running Agno examples with MLflow tracing.

```bash
pip install 'mlflow>=3.3' agno anthropic yfinance
```

--------------------------------

### Install MLflow and Start Server Locally

Source: https://mlflow.org/docs/latest/genai/eval-monitor/running-evaluation/multi-turn

Install the MLflow Python package using pip and then start a local MLflow server. This is a common setup for MLflow development and evaluation.

```shell
pip install --upgrade mlflow  

mlflow server
```

--------------------------------

### Start MLflow Tracking Server with GenAI Extra using uv

Source: https://mlflow.org/docs/latest/genai/tracing/token-usage-cost

Start the MLflow Tracking Server with the '[genai]' extra installed using 'uv'. This command assumes 'uv' installation.

```bash
uv run mlflow server
```

--------------------------------

### Install and Start MLflow Server

Source: https://mlflow.org/docs/latest/genai/governance/ai-gateway/quickstart

Installs MLflow with GenAI dependencies and starts the tracking server. The AI Gateway is built into the MLflow Tracking Server and is accessible at http://localhost:5000. Requires a SQL-based backend store and FastAPI.

```bash
pip install 'mlflow[genai]'

mlflow server --port 5000
```

--------------------------------

### Install and Use MLflow Test Plugin

Source: https://mlflow.org/docs/latest/plugins.html

Clone the MLflow repository and install the example test plugin. Then, use the plugin with a custom tracking URI scheme to log data and view it with the MLflow UI.

```bash
git clone https://github.com/mlflow/mlflow

cd mlflow

pip install -e . tests/resources/mlflow-test-plugin
```

```bash
MLFLOW_TRACKING_URI=file-plugin:$(PWD)/mlruns python examples/quickstart/mlflow_tracking.py

mlflow server --backend-store-uri ./mlruns
```

--------------------------------

### Start and End MLflow Run Example

Source: https://mlflow.org/docs/latest/api_reference/_modules/mlflow/tracking/fluent.html

Demonstrates how to start an MLflow run, access its details, end the run, and verify its status. This is useful for basic experiment tracking workflows.

```python
import mlflow

# Start run and get status
mlflow.start_run()
run = mlflow.active_run()
print(f"run_id: {run.info.run_id}; status: {run.info.status}")

# End run and get status
mlflow.end_run()
run = mlflow.get_run(run.info.run_id)
print(f"run_id: {run.info.run_id}; status: {run.info.status}")
print("--")

# Check for any active runs
print(f"Active run: {mlflow.active_run()}")
```

--------------------------------

### Run Codex CLI Setup Command

Source: https://mlflow.org/docs/latest/genai/tracing/integrations/listing/codex

Execute the `mlflow-codex setup` command to configure MLflow tracing interactively. This command prompts for installation location, MLflow tracking URI, and experiment ID, then creates necessary configuration files.

```text
~/my-project $ mlflow-codex setup  

  Configure MLflow tracing for Codex CLI  
  ? Where should MLflow tracing be installed?  
  
  ● Project  ./.codex/  (default)  
  ○ User  ~/.codex/  
  
  ↑/↓ to move, enter to select  
  
  ✓ Created /path/to/project/.codex/config.toml with notify hook  
  
  MLflow tracking URI [http://localhost:5000] http://localhost:5000  
  MLflow experiment ID [0] 1  
  
  ✓ Wrote tracing config to /path/to/project/.codex/mlflow-tracing.json  
  
  Next steps  
  1. Start the MLflow tracking server in a separate terminal:  
       mlflow server --port 5000  
  2. Launch codex — traces appear at http://localhost:5000 after each turn.  
```

--------------------------------

### RetrievalRelevance Scorer - Direct Usage

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

This example demonstrates how to directly invoke the RetrievalRelevance scorer with a trace to get feedback on context relevance.

```APIDOC
## RetrievalRelevance

### Description
Retrieval relevance measures whether each chunk is relevant to the input request. You can invoke the scorer directly with a single input for testing, or pass it to mlflow.genai.evaluate for running full evaluation on a dataset.

### Parameters
- **name** (str) - Optional - The name of the scorer. Defaults to “retrieval_relevance”.
- **model** (str | None) - Optional - Judge model to use. Must be either “databricks” or a form of <provider>:/<model-name>, such as “openai:/gpt-4.1-mini”, “anthropic:/claude-3.5-sonnet-20240620”. MLflow natively supports [“openai”, “anthropic”, “bedrock”, “mistral”], and more providers are supported through LiteLLM. Default model depends on the `MLFLOW_GENAI_JUDGE_DEFAULT_MODEL` environment variable and the tracking URI setup.
- **inference_params** (dict[str, typing.Any] | None) - Optional dictionary of inference parameters (e.g., temperature, top_p, max_tokens) to pass to the judge model for fine-grained control.
- **description** (str) - Optional - Description of the scorer. Defaults to 'Evaluate whether each retrieved context chunk is relevant to the input request.'
- **required_columns** (set[str]) - Optional - Set of required columns for the scorer. Defaults to {'inputs', 'trace'}.

### Request Example
```python
import mlflow
from mlflow.genai.scorers import RetrievalRelevance

trace = mlflow.get_trace("<your-trace-id>")
feedbacks = RetrievalRelevance(
    name="my_retrieval_relevance",
    inference_params={"temperature": 0.0},
)(trace=trace)
print(feedbacks)
```
```

--------------------------------

### General Dataset Search Examples

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Provides examples for searching datasets with various filters, including maximum results, experiment IDs, and tag-based filtering with ordering. Demonstrates iterating through paginated results.

```python
from mlflow.genai.datasets import search_datasets

# WARNING: This returns ALL datasets - use with caution!
# all_datasets = search_datasets()  # May be slow or crash

# Better: Always use filters or limits
recent_datasets = search_datasets(max_results=100)

# Search in specific experiments
exp_datasets = search_datasets(experiment_ids=["1", "2", "3"])

# Find production datasets
prod_datasets = search_datasets(
    filter_string="tags.environment = 'production'", order_by=["name ASC"]
)

# Iterate through results (pagination handled automatically)
for dataset in prod_datasets:
    print(f"{dataset.name} (ID: {dataset.dataset_id})")
    print(f"  Tags: {dataset.tags}")
```

--------------------------------

### Run Qwen Code Setup Command

Source: https://mlflow.org/docs/latest/genai/tracing/integrations/listing/qwen_code

Execute the setup command to configure MLflow tracing for Qwen Code. This interactive flow registers a stop hook and writes tracing configuration to a JSON file.

```text
~/my-project $ mlflow-qwen-code setup  

  Configure MLflow tracing for Qwen Code  
  ? Where should MLflow tracing be installed?  
  
  ● Project  ./.qwen/  (default)  
  ○ User  ~/.qwen/  
  
  ↑/↓ to move, enter to select  
  
  ✓ Registered Stop hook in /path/to/project/.qwen/settings.json  
  
  
  MLflow tracking URI [http://localhost:5000] http://localhost:5000  
  MLflow experiment ID [0] 1  
  
  ✓ Wrote tracing config to /path/to/project/.qwen/mlflow-tracing.json  
  
  Next steps  
  
  1. Start the MLflow tracking server in a separate terminal:  
       mlflow server --port 5000  
       
  2. Launch qwen — traces appear at http://localhost:5000 after each turn.  

```

--------------------------------

### Start MLflow GenAI Scorer with Sampling Configuration

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Shows how to start a registered MLflow GenAI scorer for automatic trace evaluation. You can specify a sampling rate and an optional filter string to control which traces are evaluated. The scorer must be registered before starting.

```python
import mlflow
from mlflow.genai.scorers import RelevanceToQuery, ScorerSamplingConfig

# Start scorer with 50% sampling rate
mlflow.set_experiment("my_genai_app")
scorer = RelevanceToQuery().register()
active_scorer = scorer.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))
print(f"Scorer is evaluating {active_scorer.sample_rate * 100}% of traces")

# Start scorer with filter to only evaluate specific traces
filtered_scorer = scorer.start(
    sampling_config=ScorerSamplingConfig(
        sample_rate=1.0, filter_string="YOUR_FILTER_STRING"
    )
)

```

--------------------------------

### Create a Dataset

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Creates a new dataset with a specified name and experiment ID. This is a minimal example without additional tags.

```python
simple_dataset = create_dataset(
    name="quick_test_dataset",
    experiment_id="3",  # Specific experiment
)
```

--------------------------------

### Example requirements.in file

Source: https://mlflow.org/docs/latest/self-hosting/security/secure-installs

A sample requirements.in file pinning direct dependencies.

```text
mlflow==3.10.1
```

--------------------------------

### Exact Match Scorer Example

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Instantiate and use the ExactMatch scorer to check for exact string matches between the model output and the expected output. No special setup is required beyond importing the class.

```python
from mlflow.genai.scorers.ragas import ExactMatch

scorer = ExactMatch()
feedback = scorer(
    outputs="Paris",
    expectations={"expected_output": "Paris"},
)
```

--------------------------------

### Load Serving Input Example from Model Directory

Source: https://mlflow.org/docs/latest/api_reference/_modules/mlflow/models/utils.html

Loads the serving input example from a specified model directory. Returns None if no serving input example is found or if the relevant metadata is missing.

```python
def _load_serving_input_example(mlflow_model: Model, path: str) -> str | None:
    """
    Load serving input example from a model directory. Returns None if there is no serving input
    example.

    Args:
        mlflow_model: Model metadata.
        path: Path to the model directory.

    Returns:
        Serving input example or None if the model has no serving input example.
    """
    if mlflow_model.saved_input_example_info is None:
        return None
    serving_input_path = mlflow_model.saved_input_example_info.get(SERVING_INPUT_PATH)
    if serving_input_path is None:
        return None
    with open(os.path.join(path, serving_input_path)) as handle:
        return handle.read()
```

--------------------------------

### MLflow GenAI Evaluation Example

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Demonstrates how to use MLflow's genai.evaluate function with a list of custom scorers.

```python
mlflow.genai.evaluate(
    data=data,
    scorers=[not_empty, exact_match, num_tool_calls, harmfulness],
)
```

--------------------------------

### Start Time Management

Source: https://mlflow.org/docs/latest/api_reference/java_api/org/mlflow/api/proto/Service.CreateRun.Builder.html

Methods for setting, getting, and clearing the start time of a run.

```APIDOC
## hasStartTime
### Description
Checks if the start time has been set.
### Method Signature
public boolean hasStartTime()
### Returns
True if the startTime field is set, false otherwise.
```

```APIDOC
## getStartTime
### Description
Retrieves the start time of the run.
### Method Signature
public long getStartTime()
### Returns
The start time in milliseconds.
```

```APIDOC
## setStartTime
### Description
Sets the start time of the run.
### Method Signature
public Service.CreateRun.Builder setStartTime(long value)
### Parameters
* `value` (long) - The startTime to set in milliseconds.
### Returns
This builder for chaining.
```

```APIDOC
## clearStartTime
### Description
Clears the start time of the run.
### Method Signature
public Service.CreateRun.Builder clearStartTime()
### Returns
This builder for chaining.
```

--------------------------------

### MLflow GenAI Dataset Search Examples

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Demonstrates various ways to use `search_datasets`, including fetching recent datasets, searching within specific experiments, and filtering for production datasets with sorting.

```python
from mlflow.genai.datasets import search_datasets

# WARNING: This returns ALL datasets - use with caution!
# all_datasets = search_datasets()  # May be slow or crash

# Better: Always use filters or limits
recent_datasets = search_datasets(max_results=100)

# Search in specific experiments
exp_datasets = search_datasets(experiment_ids=["1", "2", "3"])

# Find production datasets
prod_datasets = search_datasets(
    filter_string="tags.environment = 'production'", order_by=["name ASC"]
)
```

--------------------------------

### Original Prompt Example

Source: https://mlflow.org/docs/latest/genai/prompt-registry/rewrite-prompts

This is an example of a prompt before optimization.

```text
Classify the sentiment. Answer 'positive'  
or 'negative' or 'neutral'.  
  
Text: {{text}}  

```

--------------------------------

### Manual Installation of n8n MLflow Node

Source: https://mlflow.org/docs/latest/genai/tracing/integrations/listing/n8n

Installs the n8n-nodes-mlflow package manually within an existing n8n installation and starts the n8n service.

```bash
cd ~/.n8n
npm install n8n-nodes-mlflow
n8n start
```

--------------------------------

### Start Scoring

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Starts registered scoring with a specified sampling configuration, activating automatic trace evaluation.

```APIDOC
## start

### Description
Start registered scoring with the specified sampling configuration.
This method activates automatic trace evaluation for the scorer. The scorer will evaluate traces based on the provided sampling configuration, including the sample rate and optional filter criteria.

### Parameters

* **name** – Optional scorer name. If not provided, uses the scorer’s registered name or default name.
* **experiment_id** – The ID of the MLflow experiment containing the scorer. If None, uses the currently active experiment.
* **sampling_config** – Configuration object containing: - sample_rate: Fraction of traces to evaluate (0.0 to 1.0). Required. - filter_string: Optional MLflow search_traces compatible filter string.

### Returns
A new Scorer instance with updated sampling configuration.

### Example
```python
import mlflow
from mlflow.genai.scorers import RelevanceToQuery, ScorerSamplingConfig

# Start scorer with 50% sampling rate
mlflow.set_experiment("my_genai_app")
scorer = RelevanceToQuery().register()
active_scorer = scorer.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))
print(f"Scorer is evaluating {active_scorer.sample_rate * 100}% of traces")

# Start scorer with filter to only evaluate specific traces
filtered_scorer = scorer.start(
    sampling_config=ScorerSamplingConfig(
        sample_rate=1.0, filter_string="YOUR_FILTER_STRING"
    )
)
```
```

--------------------------------

### Initialize and Use TopicAdherence Scorer

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Initializes the TopicAdherence scorer and applies it to a trace. This example shows how to evaluate if the AI system adheres to specified topics during an interaction.

```python
from mlflow.genai.scorers.ragas import TopicAdherence

scorer = TopicAdherence()
feedback = scorer(
    trace=trace,
    expectations={
        "reference_topics": ["machine learning", "data science"],
    },
)
```

--------------------------------

### Register, Start, and Stop MLflow GenAI Scorer

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Demonstrates how to register a scorer, start it with a specific sampling rate, and then stop it. The scorer remains registered for later use.

```python
import mlflow
from mlflow.genai.scorers import RelevanceToQuery, ScorerSamplingConfig

# Start and then stop a scorer
mlflow.set_experiment("my_genai_app")
scorer = RelevanceToQuery().register()
active_scorer = scorer.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))
print(f"Scorer is active: {active_scorer.sample_rate > 0}")

# Stop the scorer
stopped_scorer = active_scorer.stop()
print(f"Scorer is active: {stopped_scorer.sample_rate > 0}")

# The scorer remains registered and can be restarted later
restarted_scorer = stopped_scorer.start(
    sampling_config=ScorerSamplingConfig(sample_rate=0.3)
)

```

--------------------------------

### Create a Dataset Linked to Multiple Experiments

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

This example demonstrates creating a dataset that is associated with several experiments simultaneously by providing a list of experiment IDs. Tags can be used to describe the dataset's coverage and development status.

```python
# Create a dataset linked to multiple experiments
multi_exp_dataset = create_dataset(
    name="cross_team_eval_dataset",
    experiment_id=["1", "2", "5"],  # Multiple experiment IDs
    tags={
        "coverage": "comprehensive",
        "status": "development",
    },
)
```

--------------------------------

### Set Up MLflow Server with Docker Compose

Source: https://mlflow.org/docs/latest/genai/eval-monitor/running-evaluation/multi-turn

Clone the MLflow repository, navigate to the docker-compose directory, copy the example environment file, and start the MLflow server with PostgreSQL and MinIO using Docker Compose.

```shell
git clone --depth 1 --filter=blob:none --sparse https://github.com/mlflow/mlflow.git  

cd mlflow  

git sparse-checkout set docker-compose  

cd docker-compose  

cp .env.dev.example .env  

docker compose up -d
```

--------------------------------

### Get Model Input Example

Source: https://mlflow.org/docs/latest/api_reference/_modules/mlflow/pyfunc.html

Retrieves the input example associated with the MLflow PyFunc model. This example is used for schema inference and validation.

```python
input_example = model.input_example
```

--------------------------------

### Optimize Prompts with Built-in Scorers

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

This example demonstrates how to optimize prompts using a predefined predict function, training data, prompt URIs, a GepaPromptOptimizer, and the built-in Correctness scorer. MLflow tracking is enabled by default.

```python
import mlflow
import openai
from mlflow.genai.optimize.optimizers import GepaPromptOptimizer
from mlflow.genai.scorers import Correctness

prompt = mlflow.genai.register_prompt(
    name="qa",
    template="Answer the following question: {{question}}",
)


def predict_fn(question: str) -> str:
    completion = openai.OpenAI().chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt.format(question=question)}],
    )
    return completion.choices[0].message.content


dataset = [
    {"inputs": {"question": "What is the capital of France?"}, "outputs": "Paris"},
    {"inputs": {"question": "What is the capital of Germany?"}, "outputs": "Berlin"},
]

result = mlflow.genai.optimize_prompts(
    predict_fn=predict_fn,
    train_data=dataset,
    prompt_uris=[prompt.uri],
    optimizer=GepaPromptOptimizer(reflection_model="openai:/gpt-4o"),
    scorers=[Correctness(model="openai:/gpt-4o")],
)

print(result.optimized_prompts[0].template)

```

--------------------------------

### Initialize and Use ContextRelevance Scorer

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Demonstrates how to initialize the ContextRelevance scorer with a specified model and then use it to get feedback from a trace. Ensure the 'trace' object is defined and populated before use.

```python
from mlflow.genai.scorers.trulens import ContextRelevance

scorer = ContextRelevance(model="openai:/gpt-5")
feedback = scorer(trace=trace)
```

--------------------------------

### Get LlamaIndex Version

Source: https://mlflow.org/docs/latest/api_reference/_modules/mlflow/llama_index/model.html

Retrieves the installed version of the 'llama-index.core' library. If the library is not installed, it raises an `MlflowException` prompting the user to install it.

```python
def _get_llama_index_version() -> str:
    try:
        import llama_index.core

        return llama_index.core.__version__
    except ImportError:
        raise MlflowException(
            "The llama_index module is not installed. "
            "Please install it via `pip install llama-index`."
        )
```

--------------------------------

### Optimize Prompts with Custom Scorers and Aggregation

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

This example shows how to optimize prompts using custom scorers (accuracy and brevity) and a custom aggregation function (weighted_objective). It utilizes the GepaPromptOptimizer and requires the predict_fn and dataset to be defined.

```python
import mlflow
from mlflow.genai.optimize.optimizers import GepaPromptOptimizer
from mlflow.genai.scorers import scorer


# Define custom scorers
@scorer(name="accuracy")
def accuracy_scorer(outputs, expectations):
    return 1.0 if outputs.lower() == expectations.lower() else 0.0


@scorer(name="brevity")
def brevity_scorer(outputs):
    # Prefer shorter outputs (max 50 chars gets score of 1.0)
    return min(1.0, 50 / max(len(outputs), 1))


# Define objective to combine scores
def weighted_objective(scores):
    return 0.7 * scores["accuracy"] + 0.3 * scores["brevity"]


result = mlflow.genai.optimize_prompts(
    predict_fn=predict_fn,
    train_data=dataset,
    prompt_uris=[prompt.uri],
    optimizer=GepaPromptOptimizer(reflection_model="openai:/gpt-4o"),
    scorers=[accuracy_scorer, brevity_scorer],
    aggregation=weighted_objective,
)

```

--------------------------------

### Correctness Scorer - Direct Usage

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

This example demonstrates how to directly use the `Correctness` scorer to evaluate a single input-output pair against expected facts.

```APIDOC
## Direct Usage of Correctness Scorer

### Description
This example shows how to instantiate and use the `Correctness` scorer directly to evaluate a specific model output against provided expectations.

### Method Signature
`Correctness(name: str = 'correctness', ...)`

### Parameters
- **inputs** (dict) - Required - The input provided to the model.
- **outputs** (str) - Required - The model's response.
- **expectations** (list[dict]) - Required - A list of dictionaries, where each dictionary contains an `expected_response` key.

### Example
```python
import mlflow
from mlflow.genai.scorers import Correctness

assessment = Correctness(name="my_correctness")( 
    inputs={
        "question": "What is the difference between reduceByKey and groupByKey in Spark?"
    },
    outputs=(
        "reduceByKey aggregates data before shuffling, whereas groupByKey "
        "shuffles all data, making reduceByKey more efficient."
    ),
    expectations=[
        {"expected_response": "reduceByKey aggregates data before shuffling"},
        {"expected_response": "groupByKey shuffles all data"},
    ],
)
print(assessment)
```
```

--------------------------------

### Install MLflow and Groq

Source: https://mlflow.org/docs/latest/genai/tracing/integrations/listing/groq

Install the necessary libraries for MLflow tracing and Groq integration.

```bash
pip install mlflow groq  
```

--------------------------------

### Load Serving Example from Model URI or Path

Source: https://mlflow.org/docs/latest/api_reference/_modules/mlflow/models/utils.html

Loads the serving input example from a model directory or URI. This function acts as a wrapper to read specific file content.

```python
def load_serving_example(model_uri_or_path: str):
    """
    Load serving input example from a model directory or URI.

    Args:
        model_uri_or_path: Model URI or path to the `model` directory.
            e.g. models://<model_name>/<model_version> or /path/to/model
    """
    return _read_file_content(model_uri_or_path, SERVING_INPUT_FILENAME)
```

--------------------------------

### Install MLflow and OpenAI

Source: https://mlflow.org/docs/latest/genai/eval-monitor/scorers/custom/code-examples

Update MLflow to the latest version and install the OpenAI client library. This is a prerequisite for running the example applications.

```bash
pip install --upgrade mlflow openai  
```

--------------------------------

### Install MLflow and DSPy

Source: https://mlflow.org/docs/latest/genai/flavors/dspy

Install the MLflow and DSPy libraries using pip. The -U flag ensures you get the latest versions.

```shell
pip install mlflow dspy -U
```

--------------------------------

### MLflow Agent Setup

Source: https://mlflow.org/docs/latest/api_reference/cli.html

Experimental command to install MLflow skills and launch a coding agent for repository instrumentation. Use --print to output the task prompt.

```bash
mlflow agent setup [OPTIONS]
```

--------------------------------

### Get and Use MLflow GenAI Datasets

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Demonstrates how to retrieve datasets by name or ID, access their properties, convert them to pandas DataFrames, and merge new records. Ensure you have the necessary dataset configurations.

```python
from mlflow.genai.datasets import get_dataset

# Get a dataset by name
dataset = get_dataset(name="my_evaluation_dataset")

# Get a dataset by ID
dataset = get_dataset(dataset_id="d-7f2e3a9b8c1d4e5f6a7b8c9d0e1f2a3b")

# Access dataset properties
print(f"Dataset name: {dataset.name}")
print(f"Tags: {dataset.tags}")
print(f"Created by: {dataset.created_by}")

# Work with the dataset
df = dataset.to_df()  # Convert to pandas DataFrame
schema = dataset.schema  # Get auto-computed schema
profile = dataset.profile  # Get dataset statistics

# Add new records to the dataset
new_test_cases = [
    {
        "inputs": {"question": "What is MLflow?"},
        "expectations": {"accuracy": 0.95, "contains_tracking": True},
    }
]
dataset.merge_records(new_test_cases)

```

--------------------------------

### Multi-Framework Example with OpenAI and LangChain

Source: https://mlflow.org/docs/latest/genai/tracing/app-instrumentation/automatic

Combine different LLM providers and frameworks, such as OpenAI and LangChain, within a single trace by enabling autologging for each. This example demonstrates using OpenAI directly for initial processing and then LangChain for structured output.

```bash
pip install --upgrade langchain langchain-openai  
```

```python
import mlflow  

import openai  

from mlflow.entities import SpanType  

from langchain_openai import ChatOpenAI  

from langchain_core.prompts import ChatPromptTemplate  

# Enable auto-tracing for both OpenAI and LangChain  
mlflow.openai.autolog()  
mlflow.langchain.autolog()  

@mlflow.trace(span_type=SpanType.CHAIN)  
def multi_provider_workflow(query: str):  
    # First, use OpenAI directly for initial processing  
    analysis = openai.OpenAI().chat.completions.create(  
        model="gpt-4o-mini",  
        messages=[  
            {"role": "system", "content": "Analyze the query and extract key topics."},  
            {"role": "user", "content": query},  
        ],  
    )  
    topics = analysis.choices[0].message.content  

    # Then use LangChain for structured processing  
    llm = ChatOpenAI(model="gpt-4o-mini")  
    prompt = ChatPromptTemplate.from_template(  
        "Based on these topics: {topics}\nGenerate a detailed response to: {query}"  
    )  
    chain = prompt | llm  
    response = chain.invoke({"topics": topics, "query": query})  

    return response  

# Run the function  
result = multi_provider_workflow("Explain quantum computing")  
```

--------------------------------

### Start MLflow Server with pip

Source: https://mlflow.org/docs/latest/genai/eval-monitor/quickstart

Install and upgrade the MLflow Python package and start a local MLflow Tracking Server. This is another method for setting up your environment.

```shell
pip install --upgrade mlflow

mlflow server
```

--------------------------------

### Start Open WebUI

Source: https://mlflow.org/docs/latest/genai/tracing/integrations/listing/open-webui

Starts the Open WebUI application. This is the interface you will interact with.

```bash
open-webui serve  
```

--------------------------------

### Optimized Prompt Template Example

Source: https://mlflow.org/docs/latest/genai/prompt-registry/rewrite-prompts

An example of an optimized prompt template designed to guide the target model (gpt-4o-mini) to match the behavior of a reference model (gpt-5).

```text
Optimized template:  
Classify the sentiment of the provided text. Your response must be one of the following:  

- 'positive'  
- 'negative'  
- 'neutral'  
  
Ensure your response is lowercase and contains only one of these three words.  

Text: {{text}}  

Guidelines:  
- 'positive': The text expresses satisfaction, happiness, or approval  
- 'negative': The text expresses dissatisfaction, anger, or disapproval  
- 'neutral': The text is factual or balanced without strong emotion  

Your response must match this exact format with no additional explanation.  
```

--------------------------------

### Summarization Scorer Example

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Instantiate and use the Summarization scorer to evaluate the quality of a document summary against the original text. No specific model is required for this basic instantiation.

```python
from mlflow.genai.scorers.phoenix import Summarization

scorer = Summarization()
feedback = scorer(
    inputs="Long document text...",
    outputs="Brief summary of the document.",
)

```

--------------------------------

### Initialize and Use Hallucination Scorer

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html

Instantiate the Hallucination scorer with a threshold and apply it to a trace to get feedback. Ensure the 'trace' object is properly defined before use.

```python
scorer = Hallucination(threshold=0.3)
feedback = scorer(trace=trace)
```

--------------------------------

### Get Pip Install MLflow Command

Source: https://mlflow.org/docs/latest/api_reference/_modules/mlflow/utils/environment.html

Returns a command to pip-install mlflow. If the MLFLOW_HOME environment variable exists, it returns a command for a development version install; otherwise, it returns a command to install a specific MLflow version.

```python
def _get_pip_install_mlflow():
    """
    Returns a command to pip-install mlflow. If the MLFLOW_HOME environment variable exists,
    returns "pip install -e {MLFLOW_HOME} 1>&2", otherwise
    "pip install mlflow=={mlflow.__version__} 1>&2".
    """
    if mlflow_home := os.environ.get("MLFLOW_HOME"):  # dev version
        return f"pip install -e {mlflow_home} 1>&2"
    else:
        return f"pip install mlflow=={VERSION} 1>&2"
```

--------------------------------

### Run Hermes Model Setup Wizard

Source: https://mlflow.org/docs/latest/genai/governance/ai-gateway/coding-agents/hermes-agent

Execute the Hermes setup wizard to configure model settings interactively.

```bash
hermes setup model
```

--------------------------------

### Print example output

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.metrics.html

Displays the string representation of an example object.

```python
    "a company that specializes in big data and machine learning solutions. MLflow is "
    "designed to address the challenges that data scientists and machine learning "
    "engineers face when developing, training, and deploying machine learning models."
},
)
print(str(example))
```

--------------------------------

### Configure MLflow Tracking URI and Start a Run

Source: https://mlflow.org/docs/latest/auth.html

Set the MLflow tracking URI and start a run. This example assumes basic authentication is configured via environment variables.

```python
import mlflow

mlflow.set_tracking_uri("https://<mlflow_tracking_uri>/")
with mlflow.start_run():
    ...
```

--------------------------------

### Start and manage trace with child spans

Source: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.client.html

Demonstrates how to use the MlflowClient to start a trace, create child spans, and end both the span and the trace. This example shows imperative span management.

```python
from mlflow import MlflowClient

client = MlflowClient()

span = client.start_trace("my_trace")

x = 2

# Create a child span
child_span = client.start_span(
    "child_span",
    trace_id=span.trace_id,
    parent_id=span.span_id,
    inputs={"x": x},
)

y = x**2

client.end_span(
    trace_id=child_span.trace_id,
    span_id=child_span.span_id,
    attributes={"factor": 2},
    outputs={"y": y},
)

client.end_trace(span.trace_id)

```

--------------------------------

### Install OpenTelemetry Packages

Source: https://mlflow.org/docs/latest/genai/tracing/integrations/listing/voltagent

Install the necessary OpenTelemetry SDK and OTLP protobuf exporter packages for tracing.

```bash
npm install @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto dotenv  
```