Evidently AI

https://docs.evidentlyai.com/
Admin
Evidently is an open-source framework (Apache 2.0) with 40M+ downloads that helps teams evaluate,...
Tokens:64,504
Snippets:921
Trust Score:10
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install required libraries

Source: https://docs.evidentlyai.com/quickstart_tracing

Install the necessary packages for tracing and evaluation.

```python
! pip install evidently
! pip install tracely 
! pip install openai
```

--------------------------------

### Install Evidently and dependencies

Source: https://docs.evidentlyai.com/examples/LLM_jury

Install the necessary packages to run LLM evaluation workflows.

```python
pip install evidently litellm 
```

--------------------------------

### Install Evidently Library

Source: https://docs.evidentlyai.com/docs/setup/cloud

Install the core library or the LLM-specific variant via pip.

```python
pip install evidently ## or pip install evidently[llm]
```

--------------------------------

### Launch Evidently UI with demo projects

Source: https://docs.evidentlyai.com/docs/setup/self-hosting

Starts the UI service pre-loaded with all available demo projects.

```text
evidently ui --demo-projects all
```

--------------------------------

### Initialize Tracing

Source: https://docs.evidentlyai.com/quickstart_tracing

Configure and start the tracing service by providing the project ID and destination address.

```python
project_id = str(project.id)

init_tracing(
 address="https://app.evidently.cloud/",
 api_key="YOUR_API_TOKEN",
 project_id=project_id,
 export_name="TRACING_DATASET"
 )
```

--------------------------------

### Install Evidently LLM package

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Install the Evidently library with LLM support enabled.

```python
pip install evidently[llm] 
```

--------------------------------

### Install Evidently via conda

Source: https://docs.evidentlyai.com/docs/setup/installation

Alternative installation method using the conda package manager from the conda-forge channel.

```sh
conda install -c conda-forge evidently
```

--------------------------------

### Launch Evidently UI service

Source: https://docs.evidentlyai.com/docs/setup/self-hosting

Commands to start the UI service with different workspace and port configurations.

```bash
evidently ui
```

```bash
evidently ui --workspace . /workspace
```

```bash
evidently ui --workspace ./workspace --port 8080
```

--------------------------------

### Install Evidently via pip

Source: https://docs.evidentlyai.com/docs/setup/installation

Standard installation method for the core Evidently library using the pip package manager.

```python
pip install evidently
```

--------------------------------

### Install Evidently Library

Source: https://docs.evidentlyai.com/quickstart_ml

Use this command to install the necessary Python package for ML evaluation.

```python
!pip install evidently
```

--------------------------------

### Create a Project in Python

Source: https://docs.evidentlyai.com/docs/platform/projects_manage

Use the create_project method to initialize a new project within a workspace. The org_id parameter is required for Evidently Cloud but optional for self-hosted installations.

```python
project = ws.create_project("My test project", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```

```python
project = ws.create_project("My test project")
project.description = "My project description"
project.save()
```

--------------------------------

### Install Tracely for OpenTelemetry

Source: https://docs.evidentlyai.com/docs/setup/installation

Installs the sister package for enabling tracing based on OpenTelemetry.

```sh
pip install tracely
```

--------------------------------

### Install Evidently LLM dependencies

Source: https://docs.evidentlyai.com/docs/setup/installation

Installs the core library along with additional dependencies required for LLM evaluation features.

```python
pip install evidently[llm]
```

--------------------------------

### Generate toy data for metric examples

Source: https://docs.evidentlyai.com/docs/library/metric_generator

Creates a pandas DataFrame and splits it into two Dataset objects for evaluation.

```python
import pandas as pd
import numpy as np
from evidently import Dataset
from evidently import DataDefinition

np.random.seed(42)

data = {
    "Age": np.random.randint(18, 60, size=30),
    "Salary": np.random.randint(30000, 120000, size=30),
    "Department": np.random.choice(["HR", "IT", "Finance", "Marketing", "Operations"], size=30),
    "YearsExperience": np.random.randint(1, 15, size=30),  
    "EducationLevel": np.random.choice(["High School", "Bachelor", "Master", "PhD"], size=30)  
}

dummy_df = pd.DataFrame(data)

eval_data_1 = Dataset.from_pandas(
    dummy_df.iloc[:15],
    data_definition=DataDefinition()
)
eval_data_2 = Dataset.from_pandas(
    dummy_df.iloc[15:],
    data_definition=DataDefinition()
)
```

--------------------------------

### BeginsWith()

Source: https://docs.evidentlyai.com/metrics/all_descriptors

Checks if the text starts with a given combination.

```APIDOC
## BeginsWith()

### Description
Checks if the text starts with a given combination. Returns True/False for every input.

### Parameters
- **prefix** (str) - Required - The prefix to check for.
- **alias** (str) - Optional - An alias for the descriptor.
- **case_sensitive** (bool) - Optional - Whether the check is case sensitive.

### Example
`BeginsWith(prefix="How")`
```

--------------------------------

### Initialize OpenAI Client

Source: https://docs.evidentlyai.com/quickstart_tracing

Sets up the OpenAI client instance using an API key.

```python
client = openai.OpenAI(api_key=openai_api_key)
```

--------------------------------

### Create sample dataset

Source: https://docs.evidentlyai.com/metrics/customize_metric

Generates a toy dataset for testing the custom metric implementation.

```python
data = {
    "Item": [f"Item_{i}" for i in range(1, 11)],
    "Quantity": np.random.randint(1, 50, size=10),
    "Sales": np.random.uniform(100, 5000, size=10).round(2),
}

df = pd.DataFrame(data)

dataset = Dataset.from_pandas(
    pd.DataFrame(df),
    data_definition=DataDefinition()
)
```

--------------------------------

### Create and run a report

Source: https://docs.evidentlyai.com/quickstart_llm

Initializes a report with text evaluations and executes it against a dataset.

```python
report = Report([
    TextEvals()
])

my_eval = report.run(eval_dataset, None)
```

--------------------------------

### Create an Evidently Project

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Create and save a new project within the connected workspace.

```python
project = ws.create_project("Regression testing example", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```

--------------------------------

### Create a reference dataset

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Initializes a pandas DataFrame with questions and reference answers for regression testing.

```python
data = [
    ["Why is the sky blue?", "The sky is blue because molecules in the air scatter blue light from the sun more than they scatter red light."],
    ["How do airplanes stay in the air?", "Airplanes stay in the air because their wings create lift by forcing air to move faster over the top of the wing than underneath, which creates lower pressure on top."],
    ["Why do we have seasons?", "We have seasons because the Earth is tilted on its axis, which causes different parts of the Earth to receive more or less sunlight throughout the year."],
    ["How do magnets work?", "Magnets work because they have a magnetic field that can attract or repel certain metals, like iron, due to the alignment of their atomic particles."],
    ["Why does the moon change shape?", "The moon changes shape, or goes through phases, because we see different portions of its illuminated half as it orbits the Earth."]
]

columns = ["question", "target_response"]

ref_data = pd.DataFrame(data, columns=columns)
```

--------------------------------

### Load Toy Dataset

Source: https://docs.evidentlyai.com/quickstart_ml

Fetches the adult dataset for demonstration purposes.

```python
adult_data = datasets.fetch_openml(name="adult", version=2, as_frame="auto")
adult = adult_data.frame
```

```python
url = "https://github.com/evidentlyai/evidently/blob/main/test_data/adults.parquet?raw=true"
adult = pd.read_parquet(url, engine='pyarrow')
```

--------------------------------

### Telemetry Event Log Examples

Source: https://docs.evidentlyai.com/faq/telemetry

JSON payloads representing different actions captured by the telemetry service.

```json
{
"_timestamp": "2023-07-07T14:08:44.332528Z",
"action": "startup",
"api_key": "s2s.5xmxpip2ax4ut5rrihfjhb.uqcoh71nviknmzp77ev6rd",
"error": null,
"eventn_ctx_event_id": "cfcc182d-5a2d-47d6-89dd-37590ec7b08a",
"extra": {},
"group_id": null,
"interface": "service_backend",
"os_name": "mac",
"os_version": "13.0.1",
"python_version": {
  "major": 3,
  "minor": 9,
  "patch": 16
},
"source_ip": "78.163.128.1",
"src": "api",
"tool_name": "evidently",
"tool_version": "0.3.3",
"user_id": "16d5bb6f-0400-4e2c-90f3-c3b31c95a1d3"
}
```

```json
{
"_timestamp": "2023-07-07T14:10:54.355143Z",
"action": "index",
"api_key": "s2s.5xmxpip2ax4ut5rrihfjhb.uqcoh71nviknmzp77ev6rd",
"error": null,
"eventn_ctx_event_id": "96029e42-d2fc-4372-a1b5-b15e4d2ec6a0",
"extra": {},
"group_id": null,
"interface": "service_backend",
"os_name": "mac",
"os_version": "13.0.1",
"python_version": {
"major": 3,
"minor": 9,
"patch": 16
},
"source_ip": "78.163.128.1",
"src": "api",
"tool_name": "evidently",
"tool_version": "0.3.3",
"user_id": "16d5bb6f-0400-4e2c-90f3-c3b31c95a1d3"
}
```

```json
{
  "_timestamp": "2023-07-07T14:08:44.687956Z",
  "action": "list_projects",
  "api_key": "s2s.5xmxpip2ax4ut5rrihfjhb.uqcoh71nviknmzp77ev6rd",
  "error": null,
  "eventn_ctx_event_id": "12ac8fe3-0396-430b-b035-e984a3ed2663",
  "extra": {
    "project_count": 1
  },
  "group_id": null,
  "interface": "service_backend",
  "os_name": "mac",
  "os_version": "13.0.1",
  "python_version": {
    "major": 3,
    "minor": 9,
    "patch": 16
  },
  "source_ip": "78.163.128.1",
  "src": "api",
  "tool_name": "evidently",
  "tool_version": "0.3.3",
  "user_id": "16d5bb6f-0400-4e2c-90f3-c3b31c95a1d3"
}
```

```json
{
"_timestamp": "2023-07-07T14:10:54.474555Z",
"action": "get_project_info",
"api_key": "s2s.5xmxpip2ax4ut5rrihfjhb.uqcoh71nviknmzp77ev6rd",
"error": null,
"eventn_ctx_event_id": "52bf5758-4b4c-4379-b2e6-0c1b123f3ce3",
"extra": {},
"group_id": null,
"interface": "service_backend",
"os_name": "mac",
"os_version": "13.0.1",
"python_version": {
 "major": 3,
 "minor": 9,
 "patch": 16
},
"source_ip": "78.163.128.1",
"src": "api",
"tool_name": "evidently",
"tool_version": "0.3.3",
"user_id": "16d5bb6f-0400-4e2c-90f3-c3b31c95a1d3"
}
```

```json
{
"_timestamp": "2023-07-07T14:08:46.260846Z",
"action": "project_dashboard",
"api_key": "s2s.5xmxpip2ax4ut5rrihfjhb.uqcoh71nviknmzp77ev6rd",
"error": null,
"eventn_ctx_event_id": "2dc109d4-f322-42de-8db2-d9ce86787b8b",
"extra": {},
"group_id": null,
"interface": "service_backend",
"os_name": "mac",
"os_version": "13.0.1",
"python_version": {
  "major": 3,
  "minor": 9,
  "patch": 16
},
"source_ip": "78.163.128.1",
"src": "api",
"tool_name": "evidently",
"tool_version": "0.3.3",
"user_id": "16d5bb6f-0400-4e2c-90f3-c3b31c95a1d3"
}
```

```json
{
"_timestamp": "2023-07-18T13:15:16.138786Z",
"action": "add_project",
"api_key": "s2s.5xmxpip2ax4ut5rrihfjhb.uqcoh71nviknmzp77ev6rd",
"error": null,
"eventn_ctx_event_id": "ac3d9bf3-8b26-406e-b781-30936c31da87",
"extra": {},
"group_id": null,
"interface": "service_backend",
"os_name": "mac",
"os_version": "13.0.1",
"python_version": {
 "major": 3,
 "minor": 9,
 "patch": 16
},
"source_ip": "88.225.219.1",
"src": "api",
"tool_name": "evidently",
"tool_version": "0.3.3",
"user_id": "16d5bb6f-0400-4e2c-90f3-c3b31c95a1d3"
}
```

--------------------------------

### Initialize tracing

Source: https://docs.evidentlyai.com/docs/platform/tracing_setup

Configure the tracing connection using the init_tracing function with your cloud credentials.

```python
from tracely import init_tracing

init_tracing(
   address="https://app.evidently.cloud/",
   api_key="YOUR_EVIDENTLY_TOKEN",
   project_id="YOUR_PROJECT_ID",
   export_name="YOUR_TRACING_DATASET_NAME",
   )
```

--------------------------------

### Create a local workspace

Source: https://docs.evidentlyai.com/docs/setup/self-hosting

Initializes a local directory to store evaluation snapshots.

```python
ws = Workspace.create("evidently_ui_workspace")
```

--------------------------------

### Connect to an Existing Project

Source: https://docs.evidentlyai.com/docs/platform/projects_manage

Retrieve an existing project instance using its unique Project ID.

```python
project = ws.get_project("PROJECT_ID")
```

--------------------------------

### Initialize Evidently Cloud Workspace

Source: https://docs.evidentlyai.com/examples/LLM_jury

Connect to an Evidently Cloud workspace using an API token and URL.

```python
ws = CloudWorkspace(token="YOUR_API_TOKEN", url="https://app.evidently.cloud")
```

--------------------------------

### Add descriptors to an existing Dataset

Source: https://docs.evidentlyai.com/docs/library/descriptors

Demonstrates initializing a Dataset first, then adding descriptors using the add_descriptors method.

```python
eval_dataset = Dataset.from_pandas(
    df,
    data_definition=DataDefinition(text_columns=["question", "answer"]),
)
```

```python
eval_dataset.add_descriptors(descriptors=[
    Sentiment("answer", alias="Sentiment"),
    TextLength("answer", alias="Length"),
    IncludesWords("answer", words_list=['sorry', 'apologize'], alias="Denials"),
])
```

--------------------------------

### Import OpenAIPrompting Descriptor

Source: https://docs.evidentlyai.com/metrics/customize_llm_judge

Import the necessary class from the evidently library.

```python
from evidently.descriptors import OpenAIPrompting
```

--------------------------------

### init_tracing()

Source: https://docs.evidentlyai.com/docs/platform/tracing_setup

Initializes the tracing provider to send traces to the Evidently Cloud platform.

```APIDOC
## init_tracing()

### Description
Initializes the tracing provider to start collecting and sending traces to the specified Evidently Cloud project.

### Parameters
- **address** (string) - Optional - Trace collector address. Defaults to https://app.evidently.cloud/.
- **api_key** (string) - Required - Evidently Cloud API key.
- **project_id** (string) - Required - Destination Project ID in Evidently Cloud.
- **export_name** (string) - Required - Tracing dataset name. Traces with the same name are grouped into a single dataset.
- **exporter_type** (string) - Optional - Trace export protocol: grpc or http.
- **as_global** (boolean) - Optional - Registers the tracing provider globally (True) or locally (False). Default: True.

### Request Example
```python
from tracely import init_tracing

init_tracing(
   address="https://app.evidently.cloud/",
   api_key="YOUR_EVIDENTLY_TOKEN",
   project_id="YOUR_PROJECT_ID",
   export_name="YOUR_TRACING_DATASET_NAME",
   )
```

--------------------------------

### ws.create_project(name, org_id=None)

Source: https://docs.evidentlyai.com/docs/platform/projects_manage

Creates a new project within the specified workspace.

```APIDOC
## ws.create_project(name, org_id=None)

### Description
Creates a new project instance within the workspace. For cloud installations, an `org_id` is required.

### Parameters
- **name** (str) - Required - The name of the project.
- **org_id** (str) - Optional - The organization ID for cloud installations.

### Example
```python
project = ws.create_project("My test project", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```
```

--------------------------------

### Configure a Report

Source: https://docs.evidentlyai.com/docs/library/evaluations_overview

Initializes a Report object with specific metrics or presets to summarize dataset characteristics.

```python
report = Report([
    DataSummaryPreset()
])
```

--------------------------------

### Split and Prepare Data

Source: https://docs.evidentlyai.com/quickstart_ml

Creates reference and production datasets by filtering based on education levels.

```python
adult_ref = adult[~adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
adult_prod = adult[adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
```

--------------------------------

### Run Dataset Evaluations

Source: https://docs.evidentlyai.com/quickstart_llm

Initialize a dataset with specific descriptors to score model answers. Use IncludesWords as a deterministic alternative if an OpenAI key is unavailable.

```python
eval_dataset = Dataset.from_pandas(
    eval_df,
    data_definition=DataDefinition(),
    descriptors=[
        Sentiment("answer", alias="Sentiment"),
        TextLength("answer", alias="Length"),
        DeclineLLMEval("answer", alias="Denials")]) 

# Or IncludesWords("answer", words_list=['sorry', 'apologize'], alias="Denials")
```

--------------------------------

### Import Cloud Workspace and Dashboard components

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Import modules for connecting to Evidently Cloud and defining monitoring dashboard panels.

```python
from evidently.ui.workspace.cloud import CloudWorkspace
```

```python
from evidently.ui.dashboards import DashboardPanelPlot
from evidently.ui.dashboards import DashboardPanelTestSuite
from evidently.ui.dashboards import DashboardPanelTestSuiteCounter
from evidently.ui.dashboards import TestSuitePanelType
from evidently.ui.dashboards import ReportFilter
from evidently.ui.dashboards import PanelValue
from evidently.ui.dashboards import PlotType
from evidently.ui.dashboards import CounterAgg
from evidently.tests.base_test import TestStatus
from evidently.renderers.html_widgets import WidgetSize
```

--------------------------------

### Import CloudWorkspace components

Source: https://docs.evidentlyai.com/examples/LLM_rag_evals

Import the necessary class to establish a connection with the Evidently Cloud platform.

```python
from evidently.ui.workspace import CloudWorkspace
```

--------------------------------

### Configure Dataset with OpenAI Descriptor

Source: https://docs.evidentlyai.com/metrics/customize_llm_judge

Initialize a dataset with the OpenAI descriptor to compute scores for a specific column.

```python
openai_prompting = Dataset.from_pandas(
    pd.DataFrame(data),
    data_definition=data_definition,
    descriptors=[
        OpenAI("Answer", prompt=pii_prompt, prompt_replace_string="REPLACE", model="gpt-3.5-turbo-instruct", 
               feature_type="num", alias="PII for Answer (by gpt3.5)"),
        
    ]
)
```

--------------------------------

### Create or manage a project

Source: https://docs.evidentlyai.com/examples/LLM_rag_evals

Create a new project within an organization or retrieve an existing one by its ID.

```python
project = ws.create_project("My project name", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()

# or project = ws.get_project("PROJECT_ID")
```

--------------------------------

### Generate Report with Data Summary

Source: https://docs.evidentlyai.com/faq/migration

Demonstrates the updated Report API workflow, including Dataset initialization and execution.

```python
eval_data = Dataset.from_pandas(
    pd.DataFrame(source_df),
    data_definition=DataDefinition()
)

report = Report([
    DataSummaryPreset()
])

my_eval = report.run(eval_data, None)
```

--------------------------------

### Create Evaluation Report

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Initializes a report with aggregate metrics and specific test conditions for validation.

```python
report = Report([
    TextEvals(),
    MaxValue(column="Length", tests=[lte(200)]),
    CategoryCount(column="Correctness", category="incorrect", tests=[eq(0)]),
    CategoryCount(column="Style", category="style-mismatched", tests=[eq(0, is_critical=False)]),
])
```

--------------------------------

### Browse and Search Projects

Source: https://docs.evidentlyai.com/docs/platform/projects_manage

List all projects in a workspace or search for a specific project by its name.

```python
ws.list_projects()
```

```python
ws.search_project("project_name")
```

--------------------------------

### Retrieve an existing project

Source: https://docs.evidentlyai.com/examples/LLM_rag_evals

Use this snippet to fetch a project that has already been created in the workspace.

```python
# project = ws.get_project("PROJECT_ID")
```

--------------------------------

### Configure LLM API Keys

Source: https://docs.evidentlyai.com/examples/LLM_jury

Set environment variables for OpenAI, Gemini, and Anthropic to enable LLM judge access.

```python
import os
os.environ["OPENAI_API_KEY"] = "YOUR KEY"
os.environ["GEMINI_API_KEY"] = "YOUR KEY"
os.environ["ANTHROPIC_API_KEY"] = "YOUR KEY"
```

--------------------------------

### Connect to a remote workspace

Source: https://docs.evidentlyai.com/docs/setup/self-hosting

Initializes a connection to a UI service running at a specific URL.

```python
workspace = RemoteWorkspace("http://localhost:8000")
```

--------------------------------

### Configure remote S3 storage

Source: https://docs.evidentlyai.com/docs/setup/self-hosting

Sets environment variables for S3-compatible storage and launches the UI service pointing to the bucket.

```text
FSSPEC_S3_ENDPOINT_URL=http://localhost:9000/
FSSPEC_S3_KEY=my_key FSSPEC_S3_SECRET=my_secret
evidently ui --workspace s3://my_bucket/workspace
```

--------------------------------

### ws.list_projects()

Source: https://docs.evidentlyai.com/docs/platform/projects_manage

Retrieves a list of all projects in the workspace.

```APIDOC
## ws.list_projects()

### Description
Returns a list of all available projects within the current workspace.

### Example
```python
ws.list_projects()
```
```

--------------------------------

### Run Multi-Column Evaluation

Source: https://docs.evidentlyai.com/metrics/customize_llm_judge

Evaluate context quality by referencing both context and question columns.

```python
eval_df.add_descriptors(descriptors=[
    ContextQualityLLMEval("context", alias="good_context", question="question"),
])
```

--------------------------------

### Import Evidently modules

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Import the core datasets, descriptors, reports, and metrics required for LLM evaluation.

```python
import pandas as pd
from evidently.future.datasets import Dataset
from evidently.future.datasets import DataDefinition
from evidently.future.datasets import Descriptor
from evidently.future.descriptors import *
from evidently.future.report import Report
from evidently.future.presets import TextEvals
from evidently.future.metrics import *
from evidently.future.tests import *

from evidently.features.llm_judge import BinaryClassificationPromptTemplate
```

--------------------------------

### Prepare Evaluation Dataset

Source: https://docs.evidentlyai.com/quickstart_llm

Create a pandas DataFrame containing questions and answers for evaluation.

```python
data = [
    ["What is the chemical symbol for gold?", "Gold chemical symbol is Au."],
    ["What is the capital of Japan?", "The capital of Japan is Tokyo."],
    ["Tell me a joke.", "Why don't programmers like nature? Too many bugs!"],
    ["When does water boil?", "Water's boiling point is 100 degrees Celsius."],
    ["Who painted the Mona Lisa?", "Leonardo da Vinci painted the Mona Lisa."],
    ["What’s the fastest animal on land?", "The cheetah is the fastest land animal, capable of running up to 75 miles per hour."],
    ["Can you help me with my math homework?", "I'm sorry, but I can't assist with homework."],
    ["How many states are there in the USA?", "USA has 50 states."],
    ["What’s the primary function of the heart?", "The primary function of the heart is to pump blood throughout the body."],
    ["Can you tell me the latest stock market trends?", "I'm sorry, but I can't provide real-time stock market trends. You might want to check a financial news website or consult a financial advisor."]
]
columns = ["question", "answer"]

eval_df = pd.DataFrame(data, columns=columns)
#eval_df.head()
```

--------------------------------

### Download a Dataset with Python

Source: https://docs.evidentlyai.com/docs/platform/datasets_workflow

Retrieve a dataset from the workspace by its ID and optionally convert it into a pandas DataFrame.

```python
eval_dataset = ws.load_dataset(dataset_id = "YOUR_DATASET_ID") 

#to create as pandas dataframe
df = eval_dataset.as_dataframe()
```

--------------------------------

### Generate and Upload Evaluation Report

Source: https://docs.evidentlyai.com/quickstart_tracing

Create a report with text evaluations, run it against the dataset, and upload the results to the workspace.

```python
report = Report([
    TextEvals()
])

my_eval = report.run(traced_data, None)

ws.add_run(project.id, my_eval, include_data=True)
```

--------------------------------

### Run Data Drift Report

Source: https://docs.evidentlyai.com/metrics/preset_data_drift

Initializes a Report with the DataDriftPreset to compare current and reference datasets.

```python
report = Report([
    DataDriftPreset(),
])

my_eval = report.run(current, ref)
```

--------------------------------

### Create synthetic evaluation dataset

Source: https://docs.evidentlyai.com/examples/LLM_rag_evals

Initializes a pandas DataFrame with questions, responses, and target ground truth answers for RAG evaluation.

```python
synthetic_data = [
    ["Why do we yawn?", "because it's a glitch in the matrix", "Due to mirror neurons."],
    ["Why do flowers bloom?", "Because of rising temperatures", "Because it gets warmer."]
]
columns = ["Question", "Response", "Target"]
synthetic_df_3 = pd.DataFrame(synthetic_data, columns=columns)
```

--------------------------------

### Run Data Drift Test Suite

Source: https://docs.evidentlyai.com/metrics/preset_data_drift

Initializes a Report with the DataDriftPreset and include_tests enabled to perform explicit pass/fail checks.

```python
report = Report([
    DataDriftPreset(),
],
include_tests=True)

my_eval = report.run(current, ref)
```

--------------------------------

### Generate and run a Report

Source: https://docs.evidentlyai.com/docs/library/descriptors

Uses the TextEvals preset to generate a summary report of descriptor statistics and distributions.

```python
report = Report([
    TextEvals()
])
my_eval = report.run(eval_dataset)
my_eval

# my_eval.json()
# ws.add_report(project.id, my_eval, include_data=True)
```

--------------------------------

### Connect to Cloud Workspace

Source: https://docs.evidentlyai.com/docs/setup/cloud

Initialize the CloudWorkspace object with your API token and the service URL.

```python
from evidently.ui.workspace import CloudWorkspace

ws = CloudWorkspace(
token="API_KEY",
url="https://app.evidently.cloud")
```

--------------------------------

### Import Evidently components

Source: https://docs.evidentlyai.com/examples/LLM_jury

Import required modules for dataset handling, reporting, and LLM evaluation descriptors.

```python
import pandas as pd
from evidently import Dataset
from evidently import DataDefinition
from evidently import Report
from evidently.presets import TextEvals
from evidently.tests import eq, is_in, not_in
from evidently.descriptors import LLMEval, TestSummary, ColumnTest
from evidently.llm.templates import BinaryClassificationPromptTemplate
from evidently.core.datasets import DatasetColumn
from evidently.descriptors import CustomColumnDescriptor

from evidently.ui.workspace import CloudWorkspace
```

--------------------------------

### Run an evaluation report

Source: https://docs.evidentlyai.com/docs/library/evaluations_overview

Executes the configured report against the provided dataset.

```python
my_eval = report.run(eval_data, None)
```

--------------------------------

### Create or Retrieve Evaluation Project

Source: https://docs.evidentlyai.com/examples/LLM_jury

Create a new project for storing evaluation results or fetch an existing one by ID.

```python
project = ws.create_project("LLM jury", org_id="YOUR ORG ID")
project.description = "Evals with different LLMs"

# or project = ws.get_project("PROJECT_ID")
```

--------------------------------

### Create a Custom Report

Source: https://docs.evidentlyai.com/docs/library/report

Define a report by listing metrics, including both dataset-level and column-level metrics with specified columns.

```python
report = Report([
    ColumnCount(), 
    ValueStats(column="target")
])

my_eval = report.run(eval_data_1, None)
my_eval
#my_eval.json
```

--------------------------------

### Configure Provider Options

Source: https://docs.evidentlyai.com/metrics/customize_llm_judge

Pass provider-specific options like API keys and rate limits directly using the options parameter.

```python
from evidently.utils.llm.wrapper import AnthropicOptions

llm_options_evals = Dataset.from_pandas(
    pd.DataFrame(data),
    data_definition=data_definition,
    descriptors=[
        NegativityLLMEval("Answer", provider="anthropic", model="claude-3-5-sonnet-20240620"),],
    options=AnthropicOptions(api_key="YOUR_KEY_HERE", rpm_limit=50))
```

--------------------------------

### Configure pandas display settings

Source: https://docs.evidentlyai.com/examples/LLM_rag_evals

Set pandas options to display full column content.

```python
pd.set_option('display.max_colwidth', None)
```

--------------------------------

### Run report with custom metric

Source: https://docs.evidentlyai.com/metrics/customize_metric

Initializes and executes a report containing the custom metric.

```python
report = Report([
    MyMaxMetric(column="Sales")
])
my_eval = report.run(dataset, None)
my_eval
```

--------------------------------

### Run and Upload Report

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Executes the report against the dataset and uploads the results to the Evidently Cloud.

```python
my_eval = report.run(eval_dataset, None)
ws.add_run(project.id, my_eval, include_data=True)
```

--------------------------------

### Define a BinaryClassificationPromptTemplate for Hallucination

Source: https://docs.evidentlyai.com/metrics/customize_llm_judge

Create a prompt template that references a {context} placeholder for comparing responses against source information.

```python
hallucination = BinaryClassificationPromptTemplate(
        pre_messages=[("system", "You are a judge which evaluates correctness of responses by comparing them to the trusted information source.")],
        criteria = """An HALLUCINATED response is any response that
        - Contradicts the information provided in the source.
        - Adds any new information not provided in the source.
        - Gives a response not based on the source, unless it's a refusal or a clarifying question.

        A FAITHFUL response is the response that
        - Correctly uses the information from the source, even if it only partially.
        - A response that declines to answer.
        - A response that asks a clarifying question.

        Source:
        =====
        {context}
        =====
        """,
        target_category="HALLUCINATED",
        non_target_category="FAITHFUL",
        uncertainty="unknown",
        include_reasoning=True,
        )
```

--------------------------------

### ws.get_project(project_id)

Source: https://docs.evidentlyai.com/docs/platform/projects_manage

Retrieves an existing project by its unique identifier.

```APIDOC
## ws.get_project(project_id)

### Description
Connects to an existing project using its unique Project ID.

### Parameters
- **project_id** (str) - Required - The unique identifier of the project.

### Example
```python
project = ws.get_project("PROJECT_ID")
```
```

--------------------------------

### Define and run a dataset-level Report

Source: https://docs.evidentlyai.com/docs/library/descriptors

Configures a Report with specific column constraints and executes it against a dataset.

```python
tests = Report([
    MinValue(column="Sentiment", tests=[gte(0)]),
    MaxValue(column="Length", tests=[lte(150)]),
    CategoryCount(column="Test result", category=False, share_tests=[lte(0.1)])
])

my_test_eval = tests.run(eval_dataset, None)
my_test_eval
# my_test_eval.json()
```

--------------------------------

### Compute Descriptors with Dataset

Source: https://docs.evidentlyai.com/faq/migration

Initializes a Dataset from a pandas DataFrame with specific column definitions and descriptor configurations.

```python
eval_data = Dataset.from_pandas(
    pd.DataFrame(df),
    data_definition=DataDefinition(
        text_columns=["question", "answer"]),
    descriptors=[
        Sentiment("answer", alias="Sentiment"),
        TextLength("answer", alias="Length"),
        IncludesWords("answer", words_list=['sorry', 'apologize'], alias="Denials"),
    ]
)
```

--------------------------------

### Import tracing and Evidently modules

Source: https://docs.evidentlyai.com/quickstart_tracing

Import the required modules for tracing events and workspace management.

```python
import os
import openai
import time
import uuid
from tracely import init_tracing
from tracely import trace_event
from tracely import create_trace_event
from evidently.ui.workspace import CloudWorkspace
```

--------------------------------

### Generate toy dataset

Source: https://docs.evidentlyai.com/metrics/customize_descriptor

Creates a sample Pandas DataFrame and converts it into an Evidently Dataset object.

```python
data = [
    ["Can fish fly?", "no", ""],
    ["Is the sky blue?", "yes", "yes"],
    ["Is milk liquid??", "yes", "yes"]
]

columns = ["question", "target_answer", "answer"]

df = pd.DataFrame(data, columns=columns)

eval_df = Dataset.from_pandas(
    df,
    data_definition=DataDefinition())
```

--------------------------------

### Importing Report components

Source: https://docs.evidentlyai.com/docs/library/report

Import the necessary Report class, metrics, and presets before defining your evaluation.

```python
from evidently import Report
from evidently.metrics import *
from evidently.presets import *
```

--------------------------------

### Run evaluation report

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Applies existing descriptors to the new dataset and logs the results to the Evidently workspace.

```python
eval_dataset_2.add_descriptors(descriptors=descriptors)
my_eval_2 = report.run(eval_dataset_2, None)
ws.add_run(project.id, my_eval_2, include_data=True)
```

--------------------------------

### Create toy Q&A dataset

Source: https://docs.evidentlyai.com/examples/LLM_judge

Define a list of lists containing questions, target responses, system responses, and manual labels for evaluation purposes.

```python
data = [
  ["Hi there, how do I reset my password?",
   "To reset your password, click on 'Forgot Password' on the login page and follow the instructions sent to your registered email.",
   "To change your password, select 'Forgot Password' on the login screen and follow the steps sent to your registered email address. If you don't receive the email, check your spam folder or contact support for assistance.",
   "incorrect", "adds new information (contact support)"],

  ["Where can I find my transaction history?",
   "You can view your transaction history by logging into your account and navigating to the 'Transaction History' section. Here, you can see all your past transactions. You can also filter the transactions by date or type for easier viewing.",
   "Log into your account and go to 'Transaction History' to see all your past transactions. In this section, you can view and filter your transactions by date or type. This allows you to find specific transactions quickly and easily.",
   "correct", ""],

  ["How do I add another user to my account?",
   "I am afraid it is not currently possible to add multiple users to the account. Our system supports only one user per account for security reasons. We recommend creating separate accounts for different users.",
   "To add a secondary user, go to 'Account Settings', select 'Manage Users', and enter the details of the person you want to add. You can set permissions for their access, deciding what they can and cannot do within the account.",
   "incorrect", "contradiction (incorrect answer)"],

  ["Is it possible to link multiple bank accounts?",
   "Yes, you can link multiple bank accounts by going to 'Account Settings' in the menu and selecting 'Add Bank Account'. Follow the prompts to add your bank account details. Make sure to verify each bank account by following the verification process.",
   "You can add multiple bank accounts by visiting 'Accounts' in the menu and choosing 'Add Bank Account'. Enter your bank details as prompted and complete the verification process for each account to link them successfully.",
   "incorrect", "contradiction (incorrect menu item)"],

  ["Can I use your service for cryptocurrency transactions?",
   "Currently, our service does not support cryptocurrency transactions. Please check our website for updates regarding this feature. We are continuously working to expand our services and may include this in the future.",
   "Currently, our service does not support cryptocurrency transactions. Please check our website for updates regarding this feature. We are continuously working to expand our services and may include this in the future.",
   "correct", ""]]
```

--------------------------------

### Generate an evaluation report

Source: https://docs.evidentlyai.com/examples/LLM_rag_evals

Runs a report on the prepared dataset to visualize evaluation metrics in a notebook or export them as a dictionary.

```python
report = Report([
    TextEvals()
])

my_eval = report.run(context_based_evals, None)
my_eval
```

--------------------------------

### Creating a custom report with specific metrics

Source: https://docs.evidentlyai.com/docs/library/descriptors

Build a report using individual metrics like MeanValue for granular control over descriptor analysis.

```python
custom_report = Report([
    MeanValue(column="Length"),
    MeanValue(column="Sentiment")
])

my_custom_eval = custom_report.run(eval_dataset, None)
my_custom_eval
```

--------------------------------

### Combining multiple presets

Source: https://docs.evidentlyai.com/docs/library/report

Include multiple presets in a single report by listing them within the Report constructor.

```python
report = Report([
    DataDriftPreset(), 
    DataSummaryPreset()
])

my_eval = report.run(eval_data_1, eval_data_2)
my_eval
#my_eval.json
```

--------------------------------

### Run ClassificationPreset Test Suite

Source: https://docs.evidentlyai.com/metrics/preset_classification

Generates a test suite with pass/fail checks based on a reference dataset.

```python
report = Report([
    ClassificationPreset(),
],
include_tests=True)

my_eval = report.run(current, ref)
```

--------------------------------

### GroupBy Metric Usage

Source: https://docs.evidentlyai.com/docs/library/report

Demonstrates how to initialize and run a Report using the GroupBy metric to calculate statistics per category.

```APIDOC
## GroupBy Metric

### Description
Calculates metrics separately for different groups in the data based on a specified category column.

### Usage
```python
from evidently.metrics.group_by import GroupBy

report = Report([
    GroupBy(Metric(column="ColumnName"), "CategoryColumn"),
])
report.run(data, None)
```

### Notes
- Auto-generated Test conditions cannot be used when using GroupBy.
```

--------------------------------

### Add descriptors during Dataset creation

Source: https://docs.evidentlyai.com/docs/library/descriptors

Configures descriptors simultaneously while initializing the Dataset object.

```python
eval_dataset = Dataset.from_pandas(
    df,
    data_definition=DataDefinition(
        text_columns=["question", "answer"]),
    descriptors=[
        Sentiment("answer", alias="Sentiment"),
        TextLength("answer", alias="Length"),
        IncludesWords("answer", words_list=['sorry', 'apologize'], alias="Denials"),
    ]
)
```

--------------------------------

### Create an Evidently Dataset for RAG evaluation

Source: https://docs.evidentlyai.com/examples/LLM_rag_evals

Initializes a dataset object from a pandas dataframe, specifying text columns and LLM-based descriptors for evaluation.

```python
context_based_evals = Dataset.from_pandas(
    synthetic_df,
    data_definition=DataDefinition(
        text_columns=["Question", "Context", "Response"],
    ),
    descriptors=[
        FaithfulnessLLMEval("Response", context="Context"),
        ContextQualityLLMEval("Context", question="Question"),
    ]
)
# context_based_evals.as_dataframe()
```

--------------------------------

### Running Tests with Reference Data

Source: https://docs.evidentlyai.com/docs/library/tests

Compare current data against a reference dataset to derive test conditions.

```python
my_eval = report.run(eval_data_1, eval_data_2) # eval_data_2 is reference
```

--------------------------------

### Running Tests with Heuristics

Source: https://docs.evidentlyai.com/docs/library/tests

Execute tests using predefined rules when no reference dataset is provided.

```python
my_eval = report.run(eval_data_1, None) # no reference data
```

--------------------------------

### Preview reference data

Source: https://docs.evidentlyai.com/examples/LLM_regression_testing

Configures pandas display options to show full column content and displays the head of the reference dataset.

```python
pd.set_option('display.max_colwidth', None)
ref_data.head()
```

--------------------------------

### Initialize an Evidently Dataset object

Source: https://docs.evidentlyai.com/examples/LLM_judge

Map column types using DataDefinition and convert the pandas DataFrame into an Evidently Dataset.

```python
definition = DataDefinition(
    text_columns=["question", "target_response", "new_response"],
    categorical_columns=["label"]
    )

eval_dataset = Dataset.from_pandas(
    golden_dataset,
    data_definition=definition)
```

--------------------------------

### Define Evaluation Prompt

Source: https://docs.evidentlyai.com/metrics/customize_llm_judge

Create a prompt string with a placeholder for the text to be evaluated.

```python
pii_prompt = """
Please identify whether the below text contains personally identifiable information, such as name, address, date of birth, or other.
Text: REPLACE 
Use the following categories for PII identification:
1 if text contains PII
0 if text does not contain PII
0 if the provided data is not sufficient to make a clear determination
Return only one category.
"""
```

--------------------------------

### Combine Data Summary and Tests in a Report

Source: https://docs.evidentlyai.com/faq/migration

Shows how to include both data summary presets and specific test conditions within a single Report object.

```python
report = Report([
     DataSummaryPreset(),
     MaxValue(column="Length", tests=[lt(100)]),
])
```

--------------------------------

### Create LLM Judge Panel

Source: https://docs.evidentlyai.com/examples/LLM_jury

Initializes a dataset with multiple LLM evaluators and a summary test to aggregate approval metrics.

```python
llm_evals = Dataset.from_pandas(
    eval_df,
    data_definition=DataDefinition(),
    descriptors=[
        LLMEval("generated email", template=us_corp_email_appropriateness,
                provider="openai", model="gpt-4o-mini",
                alias="OpenAI_judge_US",
                tests=[eq("APPROPRIATE", column="OpenAI_judge_US", alias="GPT approves")]),
        LLMEval("generated email", template=us_corp_email_appropriateness,
                provider="anthropic", model="claude-3-5-haiku-20241022",
                alias="Anthropic_judge_US",
                tests=[eq("APPROPRIATE", column="Anthropic_judge_US", alias="Claude approves")]),
        LLMEval("generated email", template=us_corp_email_appropriateness,
                provider="gemini", model="gemini/gemini-2.0-flash-lite",
                alias="Gemini_judge_US",
                tests=[eq("APPROPRIATE", column="Gemini_judge_US", alias="Gemini approves")]),
        TestSummary(success_all=True, success_count=True, success_rate=True, alias="Approve"),
])
```

--------------------------------

### Generate toy data for evaluation

Source: https://docs.evidentlyai.com/metrics/customize_hf_descriptor

Create a pandas DataFrame and convert it into an Evidently Dataset object for testing.

```python
import pandas as pd

from evidently import Dataset
from evidently import DataDefinition

data = [
    ["Why is the sky blue?", 
     "The sky is blue because molecules in the air scatter blue light from the sun more than they scatter red light.", 
     "because air scatters blue light more"],
    ["How do airplanes stay in the air?", 
     "Airplanes stay in the air because their wings create lift by forcing air to move faster over the top of the wing than underneath, which creates lower pressure on top.", 
     "because wings create lift"],
    ["Why do we have seasons?", 
     "We have seasons because the Earth is tilted on its axis, which causes different parts of the Earth to receive more or less sunlight throughout the year.", 
     "because Earth is tilted"],
    ["How do magnets work?", 
     "Magnets work because they have a magnetic field that can attract or repel certain metals, like iron, due to the alignment of their atomic particles.", 
     "because of magnetic fields"],
    ["Why does the moon change shape?", 
     "The moon changes shape, or goes through phases, because we see different portions of its illuminated half as it orbits the Earth.", 
     "because it rotates"],
    ["What movie should I watch tonight?", 
     "A movie is a motion picture created to entertain, educate, or inform viewers through a combination of storytelling, visuals, and sound.", 
     "watch a movie that suits your mood"]
]

columns = ["question", "context", "response"]

df = pd.DataFrame(data, columns=columns)

eval_df = Dataset.from_pandas(
  df,
  data_definition=DataDefinition())
```

--------------------------------

### Run Regression Test Suite with reference

Source: https://docs.evidentlyai.com/metrics/preset_regression

Use this to generate pass/fail tests based on a reference dataset.

```python
report = Report([
    RegressionPreset(),
],
include_tests=True)

my_eval = report.run(current, ref)
```

--------------------------------

### Upload a Dataset with Python

Source: https://docs.evidentlyai.com/docs/platform/datasets_workflow

Use the add_dataset method to upload a pandas DataFrame as an Evidently Dataset to a specific project.

```python
eval_data = Dataset.from_pandas(
    source_df,
    data_definition=DataDefinition()
)
ws.add_dataset(
    dataset = eval_data, 
    name = "dataset_name",
    project_id = project.id, 
    description = "Optional description")
```
Tokens:
Raw