Ragas

https://docs.ragas.io/en/stable/
Admin
Ragas is an open-source library that provides a comprehensive suite of tools to evaluate AI...
Tokens:523,619
Snippets:3,627
Trust Score:8.7
Update:1 week ago
Show doc for...
Tokens:
Raw
### ragas quickstart examples

Source: https://docs.ragas.io/en/stable/howtos/cli

Examples of using `ragas quickstart`: list available templates, create a RAG evaluation project, and create a project in a specific directory.

```bash
# List available templates
ragas quickstart

# Create a RAG evaluation project
ragas quickstart rag_eval

# Create project in a specific directory
ragas quickstart rag_eval --output-dir ./my-project

```

--------------------------------

### Quick start with ragas

Source: https://docs.ragas.io/en/stable/howtos/cli

A 60-second quick start: create a project with `uvx ragas quickstart rag_eval`, install dependencies with `uv sync`, set the OpenAI API key, and run the evaluation script.

```bash
# Create project
uvx ragas quickstart rag_eval
cd rag_eval

# Install dependencies
uv sync

# Set API key
export OPENAI_API_KEY="your-key"

# Run evaluation
uv run python evals.py

```

--------------------------------

### Install ragas-examples[text2sql]

Source: https://docs.ragas.io/en/stable/howtos/applications/text2sql

Install the ragas-examples package with the text2sql extra to set up the environment for the guide.

```bash
uv pip install "ragas-examples[text2sql]"

```

--------------------------------

### Install ragas-examples package

Source: https://docs.ragas.io/en/stable/howtos/applications/benchmark_llm

Install the ragas-examples package which contains the benchmark LLM example code.

```bash
pip install ragas[examples]

```

--------------------------------

### Quickstart with uvx (Recommended)

Source: https://docs.ragas.io/en/stable/getstarted/evals

Use uvx to create a new ragas project named 'rag_eval' and sync dependencies. This is the recommended method for installing ragas.

```bash
uvx ragas quickstart rag_eval
cd rag_eval
uv sync

```

--------------------------------

### Quickstart with pip install ragas

Source: https://docs.ragas.io/en/stable/getstarted/evals

Install ragas via pip, then create a new project and sync dependencies. This is an alternative to the uvx method.

```bash
pip install ragas
ragas quickstart rag_eval
cd rag_eval
uv sync

```

--------------------------------

### Install ragas[examples] and set OPENAI_API_KEY

Source: https://docs.ragas.io/en/stable/howtos/applications/align-llm-as-judge

Install the ragas package with the examples extra and set your OpenAI API key as an environment variable. This is the setup step before running the judge alignment evaluation pipeline.

```bash
uv pip install "ragas[examples]"
export OPENAI_API_KEY="your-api-key-here"

```

--------------------------------

### Install ragas-examples[improverag] dependencies

Source: https://docs.ragas.io/en/stable/howtos/applications/evaluate-and-improve-rag

Installs the dependencies required to run the RAG example. Use this command before running the RAG app.

```bash
uv pip install "ragas-examples[improverag]"

```

--------------------------------

### Create the project with uvx or ragas

Source: https://docs.ragas.io/en/stable/howtos/cli/rag_eval

Commands to create a new RAG evaluation project using the `rag_eval` template. Use `uvx ragas quickstart rag_eval` if you don't have ragas installed, or `ragas quickstart rag_eval` if you do. After creation, change into the `rag_eval` directory.

```bash
# Using uvx (no installation required)
uvx ragas quickstart rag_eval
cd rag_eval

# Or with ragas installed
ragas quickstart rag_eval
cd rag_eval

```

--------------------------------

### Install zeno-client

Source: https://docs.ragas.io/en/stable/howtos/integrations/_zeno

Install the zeno-client package using pip.

```bash
pip install zeno-client

```

--------------------------------

### Install dependencies with uv sync

Source: https://docs.ragas.io/en/stable/getstarted/quickstart

Install the project dependencies using uv.

```bash
uv sync

```

--------------------------------

### Install Google packages

Source: https://docs.ragas.io/en/stable/extra/components/choose_generator_llm

Install langchain-google-genai and langchain-google-vertexai packages.

```bash
pip install langchain-google-genai langchain-google-vertexai

```

--------------------------------

### Call QA chain with example

Source: https://docs.ragas.io/en/stable/howtos/integrations/_langchain

Calls the QA chain with an example from the examples list and prints the result.

```python
result = qa_chain(examples[4])
result["result"]

```

--------------------------------

### Install ragas and create project with pip

Source: https://docs.ragas.io/en/stable/getstarted/quickstart

Install ragas first, then create the project and change into its directory.

```bash
pip install ragas
ragas quickstart rag_eval
cd rag_eval

```

--------------------------------

### Create the project with ragas quickstart

Source: https://docs.ragas.io/en/stable/howtos/cli/prompt_evals

Initializes the prompt_evals project and navigates into the directory.

```bash
ragas quickstart prompt_evals
cd prompt_evals

```

--------------------------------

### Get HuggingFace tokenizer

Source: https://docs.ragas.io/en/stable/references/tokenizers

Example usage to get a HuggingFace tokenizer for a specific model, such as meta-llama/Llama-2-7b.

```python
>>> # Get HuggingFace tokenizer
>>> tokenizer = get_tokenizer("huggingface", model_name="meta-llama/Llama-2-7b")
```

--------------------------------

### Create the project with ragas quickstart

Source: https://docs.ragas.io/en/stable/howtos/cli/llamaIndex_agent_evals

Initializes the `llamaIndex_agent_evals` template and changes into the project directory.

```bash
ragas quickstart llamaIndex_agent_evals
cd llamaIndex_agent_evals

```

--------------------------------

### Get tiktoken for a specific model

Source: https://docs.ragas.io/en/stable/references/tokenizers

Example usage to get a tiktoken tokenizer for a specific model, such as gpt-4.

```python
>>> # Get tiktoken for a specific model
>>> tokenizer = get_tokenizer("tiktoken", model_name="gpt-4")
```

--------------------------------

### Get default tiktoken tokenizer

Source: https://docs.ragas.io/en/stable/references/tokenizers

Example usage to get the default tiktoken tokenizer without specifying parameters.

```python
>>> # Get default tiktoken tokenizer
>>> tokenizer = get_tokenizer()
```

--------------------------------

### Create project with uvx (Recommended)

Source: https://docs.ragas.io/en/stable/getstarted/quickstart

No installation required. uvx automatically downloads and runs ragas, then creates the project directory and changes into it.

```bash
uvx ragas quickstart rag_eval
cd rag_eval

```

--------------------------------

### Project structure generated by quickstart

Source: https://docs.ragas.io/en/stable/getstarted/evals

Shows the directory layout created by the quickstart command, including key files like evals.py and rag.py.

```text
rag_eval/
├── README.md             # Project documentation and setup instructions
├── pyproject.toml        # Project configuration for uv and pip
├── evals.py              # Your evaluation workflow
├── rag.py                # Your RAG/LLM application
├── __init__.py           # Makes this a Python package
└── evals/                # Evaluation artifacts
    ├── datasets/         # Test data files (optional)
    ├── experiments/      # Results from running evaluations (CSV files saved here)
    └── logs/             # Evaluation execution logs

```

--------------------------------

### Install DSPy extra to resolve ImportError

Source: https://docs.ragas.io/en/stable/howtos/customizations/optimizers

If you get `ImportError: DSPy optimizer requires dspy-ai`, install the DSPy extra using either `uv add "ragas[dspy]"` or `pip install "ragas[dspy]"`.

```bash
# Install the DSPy extra
uv add "ragas[dspy]"
# or
pip install "ragas[dspy]"

```

--------------------------------

### ragas quickstart command syntax

Source: https://docs.ragas.io/en/stable/howtos/cli

Shows the syntax for the `ragas quickstart` command. The TEMPLATE argument is optional; leave empty to see available templates. The `-o, --output-dir` option specifies the directory to create the project in (default: current directory).

```bash
ragas quickstart [TEMPLATE] [OPTIONS]

```

--------------------------------

### Custom EvaluationManager for batch cancellation

Source: https://docs.ragas.io/en/stable/howtos/customizations/cancellation

Defines an EvaluationManager class that tracks multiple executors, allows starting evaluations, cancelling all running ones, and cleaning up completed ones. The usage example shows starting two evaluations and cancelling all.

```python
class EvaluationManager:
    def __init__(self):
        self.executors = []

    def start_evaluation(self, dataset, metrics):
        executor = evaluate(dataset=dataset, metrics=metrics, return_executor=True)
        self.executors.append(executor)
        return executor

    def cancel_all(self):
        """Cancel all running evaluations."""
        for executor in self.executors:
            if not executor.is_cancelled():
                executor.cancel()
        print(f"Cancelled {len(self.executors)} evaluations")

    def cleanup_completed(self):
        """Remove completed executors."""
        self.executors = [ex for ex in self.executors if not ex.is_cancelled()]

# Usage
manager = EvaluationManager()

# Start multiple evaluations
exec1 = manager.start_evaluation(dataset1, metrics)
exec2 = manager.start_evaluation(dataset2, metrics)

# Cancel all if needed
manager.cancel_all()

```

--------------------------------

### Example: Creating and using a RankingMetric

Source: https://docs.ragas.io/en/stable/references/metrics

Shows how to instantiate a RankingMetric with an LLM and prompt, then score a list of results. The output is a list of ranked items.

```python
>>> from ragas.metrics import RankingMetric
>>> from ragas.llms import llm_factory
>>> from openai import OpenAI
>>>
>>> # Create an LLM instance
>>> client = OpenAI(api_key="your-api-key")
>>> llm = llm_factory("gpt-4o-mini", client=client)
>>>
>>> # Create a ranking metric that returns top 3 items
>>> metric = RankingMetric(
...     name="relevance_ranking",
...     llm=llm,
...     prompt="Rank these results by relevance: {results}",
...     allowed_values=3
... )
>>>
>>> # Score with the metric
>>> result = metric.score(
...     llm=llm,
...     results="result1, result2, result3"
... )
>>> print(result.value)  # Output: a list of 3 ranked items
```

--------------------------------

### Create the benchmark_llm project

Source: https://docs.ragas.io/en/stable/howtos/cli/benchmark_llm

Initializes the benchmark_llm template and navigates into the project directory.

```bash
ragas quickstart benchmark_llm
cd benchmark_llm

```

--------------------------------

### DSPyOptimizer usage example

Source: https://docs.ragas.io/en/stable/references/optimizers

Shows how to instantiate DSPyOptimizer with num_candidates, max_bootstrapped_demos, and max_labeled_demos, then create an InstructionConfig and call optimize_prompts.

```python
from ragas.optimizers import DSPyOptimizer
from ragas.config import InstructionConfig

optimizer = DSPyOptimizer(
    num_candidates=10,
    max_bootstrapped_demos=5,
    max_labeled_demos=5,
)

config = InstructionConfig(llm=llm, optimizer=optimizer)
metric.optimize_prompts(dataset, config)

```

--------------------------------

### Workflow implementation example (workflow.py)

Source: https://docs.ragas.io/en/stable/howtos/cli/workflow_eval

Shows how to use the default workflow client to process an email and get a result with category, extracted fields, and response.

```python
from workflow import default_workflow_client

workflow = default_workflow_client()
result = workflow.process_email("I found a bug in version 2.1.4...")
# Returns: category, extracted fields, response

```

--------------------------------

### Initialize LLMs with Various Providers

Source: https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v03_to_v04

Examples of initializing different providers using the unified llm_factory.

```python
# OpenAI
llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI(api_key="..."))

# Anthropic
llm = llm_factory("claude-3-sonnet-20240229", client=AsyncAnthropic(api_key="..."))

# Google
llm = llm_factory("gemini-2.0-flash", client=...)
```

--------------------------------

### Predefined JSON object to simulate API response

Source: https://docs.ragas.io/en/stable/howtos/integrations/_langgraph_agent_evaluation

A predefined JSON object that simulates the metals.dev API response, allowing you to get started without a live API key.

```python
metal_price = {
    "gold": 88.1553,
    "silver": 1.0523,
    "platinum": 32.169,
    "palladium": 35.8252,
    "lbma_gold_am": 88.3294,
    "lbma_gold_pm": 88.2313,
    "lbma_silver": 1.0545,
    "lbma_platinum_am": 31.99,
    "lbma_platinum_pm": 32.2793,
    "lbma_palladium_am": 36.0088,
    "lbma_palladium_pm": 36.2017,
    "mcx_gold": 93.2689,
    "mcx_gold_am": 94.281,
    "mcx_gold_pm": 94.1764,
    "mcx_silver": 1.125,
    "mcx_silver_am": 1.1501,
    "mcx_silver_pm": 1.1483,
    "ibja_gold": 93.2713,
    "copper": 0.0098,
    "aluminum": 0.0026,
    "lead": 0.0021,
    "nickel": 0.0159,
    "zinc": 0.0031,
    "lme_copper": 0.0096,
    "lme_aluminum": 0.0026,
    "lme_lead": 0.002,
    "lme_nickel": 0.0158,
    "lme_zinc": 0.0031,
}

```

--------------------------------

### Initialize OpenAI LLM with llm_factory

Source: https://docs.ragas.io/en/stable/getstarted/evals

Creates an LLM instance using the llm_factory with the model 'gpt-4o'. This is already set up in the quickstart project's _init_clients() function.

```python
from ragas.llms import llm_factory

llm = llm_factory("gpt-4o")
```

--------------------------------

### NonLLMContextPrecisionWithReference example with output

Source: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision

Measures context precision using non-LLM similarity (Levenshtein distance) when both retrieved and reference contexts are available. Requires the rapidfuzz package (`pip install rapidfuzz`). The output is a similarity score close to 1.0.

```python
from ragas import SingleTurnSample
from ragas.metrics import NonLLMContextPrecisionWithReference

context_precision = NonLLMContextPrecisionWithReference()

sample = SingleTurnSample(
    retrieved_contexts=["The Eiffel Tower is located in Paris."],
    reference_contexts=["Paris is the capital of France.", "The Eiffel Tower is one of the most famous landmarks in Paris."]
)

await context_precision.single_turn_ascore(sample)
```

```text
0.9999999999
```

--------------------------------

### Usage examples for llm_factory

Source: https://docs.ragas.io/en/stable/references/llms

Examples showing basic usage, caching, Anthropic, Google Gemini with LiteLLM, explicit adapter selection, and async usage. Requires pre-initialized client instances.

```python
from openai import OpenAI
# Basic usage
client = OpenAI(api_key="...")
llm = llm_factory("gpt-4o-mini", client=client)
response = llm.generate(prompt, ResponseModel)

# With caching (recommended for experiments)
from ragas.cache import DiskCacheBackend
cache = DiskCacheBackend()
llm = llm_factory("gpt-4o-mini", client=client, cache=cache)

# Anthropic
from anthropic import Anthropic
client = Anthropic(api_key="...")
llm = llm_factory("claude-3-sonnet", provider="anthropic", client=client)

# Google Gemini (auto-detects litellm adapter)
from litellm import OpenAI as LiteLLMClient
client = LiteLLMClient(api_key="...", model="gemini-2.0-flash")
llm = llm_factory("gemini-2.0-flash", client=client)

# Explicit adapter selection
llm = llm_factory("gemini-2.0-flash", client=client, adapter="litellm")

# Async
from openai import AsyncOpenAI
client = AsyncOpenAI(api_key="...")
llm = llm_factory("gpt-4o-mini", client=client)
response = await llm.agenerate(prompt, ResponseModel)
```

--------------------------------

### Install Griptape and Ragas packages

Source: https://docs.ragas.io/en/stable/howtos/integrations/griptape

Installs the required packages for the tutorial: griptape with all extras and ragas. Use this in a Jupyter notebook or similar environment.

```bash
%pip install "griptape[all]" ragas -q

```

--------------------------------

### Calculate total cost of testset generation

Source: https://docs.ragas.io/en/stable/howtos/applications/_cost

After generating a testset, call testset.total_cost with per-token prices for input and output tokens to get the total cost. The example uses $5 per million input tokens and $15 per million output tokens, resulting in a cost of 0.20967.

```python
# total cost for the generation process
testset.total_cost(cost_per_input_token=5 / 1e6, cost_per_output_token=15 / 1e6)

```

```output
0.20967000000000002

```

--------------------------------

### Configure OCI authentication - CLI setup

Source: https://docs.ragas.io/en/stable/howtos/integrations/oci_genai

Run the OCI CLI setup command to configure authentication.

```bash
oci setup config

```

--------------------------------

### Install dependencies with pip

Source: https://docs.ragas.io/en/stable/howtos/cli/improve_rag

Alternative dependency installation using pip with editable install.

```bash
pip install -e .
```

--------------------------------

### Create the project with ragas quickstart workflow_eval

Source: https://docs.ragas.io/en/stable/howtos/cli/workflow_eval

Initializes a new project using the `workflow_eval` template and changes into the project directory.

```bash
ragas quickstart workflow_eval
cd workflow_eval

```

--------------------------------

### Create a custom LLM with system prompt using llm_factory

Source: https://docs.ragas.io/en/stable/howtos/customizations/customize_models

Use `llm_factory` to create a custom LLM instance with a system prompt. The example uses OpenAI's GPT-4o and requires an OpenAI API key. System prompts are useful for fine-tuned models that expect specific instructions, guiding evaluation behavior consistently, or models that require custom prompts.

```python
from ragas.llms import llm_factory
from openai import OpenAI

client = OpenAI(api_key="your-key")
llm = llm_factory(
    "gpt-4o",
    client=client,
    system_prompt="You are a helpful assistant that evaluates RAG systems."
)

```

--------------------------------

### Install prerequisite packages

Source: https://docs.ragas.io/en/stable/howtos/integrations/amazon_bedrock

Installs required Python packages for the tutorial, including boto3, opensearch-py, botocore, awscli, retrying, ragas, and langchain-aws.

```bash
%pip install --upgrade -q boto3 opensearch-py botocore awscli retrying ragas langchain-aws

```

--------------------------------

### Set up environment and Helicone configuration

Source: https://docs.ragas.io/en/stable/howtos/integrations/_helicone

Sets up the environment by importing necessary modules, configuring the Helicone API key, and setting the OpenAI API key. It also verifies that the Helicone API key has been replaced with a real value.

```python
import os

from datasets import Dataset

from ragas import evaluate
from ragas.integrations.helicone import helicone_config  # import helicone_config
from ragas.metrics import answer_relevancy, context_precision, faithfulness

# Set up Helicone
HELICONE_API_KEY = (
    "your_helicone_api_key_here"  # Replace with your actual Helicone API key
)
helicone_config.api_key = HELICONE_API_KEY
os.environ["OPENAI_API_KEY"] = (
    "your_openai_api_key_here"  # Replace with your actual OpenAI API key
)

# Verify Helicone API key is set
if HELICONE_API_KEY == "your_helicone_api_key_here":
    raise ValueError(
        "Please replace 'your_helicone_api_key_here' with your actual Helicone API key."
    )

```

--------------------------------

### Install dependencies with pip

Source: https://docs.ragas.io/en/stable/getstarted/quickstart

Install the project dependencies using pip.

```bash
pip install -e .

```

--------------------------------

### Install langchain-aws package

Source: https://docs.ragas.io/en/stable/extra/components/choose_generator_llm

Install the langchain-aws package via pip.

```bash
pip install langchain-aws

```

--------------------------------

### Groq with llm_factory

Source: https://docs.ragas.io/en/stable/howtos/llm-adapters

Create a Groq client and pass it to llm_factory with provider="groq". The Instructor adapter is used automatically.

```python
from groq import Groq
from ragas.llms import llm_factory

client = Groq(api_key="your-key")
llm = llm_factory("mixtral-8x7b", provider="groq", client=client)
# Uses Instructor adapter automatically

```

--------------------------------

### GeneticOptimizer usage example

Source: https://docs.ragas.io/en/stable/references/optimizers

Shows how to instantiate GeneticOptimizer with max_steps and population_size, then create an InstructionConfig and call optimize_prompts on a metric.

```python
from ragas.optimizers import GeneticOptimizer
from ragas.config import InstructionConfig

optimizer = GeneticOptimizer(
    max_steps=50,
    population_size=10,
)

config = InstructionConfig(llm=llm, optimizer=optimizer)
metric.optimize_prompts(dataset, config)

```

--------------------------------

### Install langchain-openai package

Source: https://docs.ragas.io/en/stable/extra/components/choose_generator_llm

Install the langchain-openai package via pip.

```bash
pip install langchain-openai

```

--------------------------------

### Example output of reference

Source: https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_testgen-custom-single-hop

Shows an example of the generated reference answer.

```text
'Yes'
```

--------------------------------

### Example output of user_input

Source: https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_testgen-custom-single-hop

Shows an example of a generated Yes/No question.

```text
'Does the Diversity, Inclusion & Belonging (DIB) Team at GitLab have a structured approach to encourage collaborations among team members through various communication methods?'
```

--------------------------------

### Run the RAG system example

Source: https://docs.ragas.io/en/stable/tutorials/rag

Command to run the simple RAG system example from the ragas_examples package.

```bash
python -m ragas_examples.rag_eval.rag
```

--------------------------------

### Install Git LFS

Source: https://docs.ragas.io/en/stable/howtos/integrations/_arize

Installs Git LFS, which is used to download the dataset.

```bash
!git lfs install

```

--------------------------------

### Create the text2sql project

Source: https://docs.ragas.io/en/stable/howtos/cli/text2sql

Initializes the text2sql template and changes into the project directory.

```bash
ragas quickstart text2sql
cd text2sql

```

--------------------------------

### Install Dependencies

Source: https://docs.ragas.io/en/stable/howtos/applications/vertexai_alignment

Install required packages: langchain-core, langchain-google-vertexai, langchain, and ragas.

```bash
%pip install --upgrade --user --quiet langchain-core langchain-google-vertexai langchain ragas

```
Tokens:
Raw