### Install nervaluate

Source: https://github.com/mantisai/nervaluate/blob/main/README.md

Install the package via pip.

```bash
pip install nervaluate
```

--------------------------------

### Complete NER Pipeline Example

Source: https://context7.com/mantisai/nervaluate/llms.txt

Provides a full example of a Named Entity Recognition pipeline, from simulated model predictions (BIO tags) to evaluation using the `Evaluator` class. This demonstrates a typical workflow.

```python
from nervaluate.evaluator import Evaluator
from nervaluate.utils import collect_named_entities

# Simulated NER model output (list of BIO tags per sentence)
# In practice, these would come from your NER model's predict() method
y_true = [
    ['O', 'B-PER', 'I-PER', 'O', 'O', 'B-ORG', 'I-ORG'],
    ['B-LOC', 'O', 'O', 'B-PER', 'O'],
    ['O', 'B-DATE', 'I-DATE', 'O', 'B-ORG'],
]

y_pred = [
    ['O', 'B-PER', 'I-PER', 'O', 'O', 'B-ORG', 'I-ORG'],  # Perfect match
    ['B-LOC', 'I-LOC', 'O', 'B-PER', 'O'],                 # LOC extended incorrectly
    ['O', 'B-DATE', 'O', 'O', 'B-ORG'],                    # DATE truncated
]


```

--------------------------------

### Generate NER Summary Report (Overall and Per-Entity)

Source: https://context7.com/mantisai/nervaluate/llms.txt

Shows how to generate formatted summary reports for NER evaluation. The first example displays overall metrics across all scenarios, while the second breaks down metrics per entity type for a specific scenario ('strict').

```python
from nervaluate.evaluator import Evaluator

true = [
    ['O', 'B-PER', 'I-PER', 'O', 'O', 'O', 'B-ORG', 'I-ORG'],
    ['O', 'B-LOC', 'B-PER', 'I-PER', 'O', 'O', 'B-DATE'],
]

pred = [
    ['O', 'O', 'B-PER', 'I-PER', 'O', 'O', 'B-ORG', 'I-ORG'],
    ['O', 'B-LOC', 'I-LOC', 'B-PER', 'O', 'O', 'B-DATE'],
]

evaluator = Evaluator(true, pred, tags=['PER', 'ORG', 'LOC', 'DATE'], loader="list")

# Overall report showing all scenarios
print(evaluator.summary_report(mode='overall'))

# Per-entity breakdown for strict scenario
print(evaluator.summary_report(mode='entities', scenario='strict'))
```

--------------------------------

### Load Dictionary/Span Format Data

Source: https://context7.com/mantisai/nervaluate/llms.txt

Use the DictLoader for pre-computed entity spans defined as dictionaries with label, start, and end keys.

```python
from nervaluate.evaluator import Evaluator

# Dictionary format: list of documents, each containing list of entity spans
true_spans = [
    [
        {"label": "PER", "start": 1, "end": 2},   # John Smith
        {"label": "ORG", "start": 5, "end": 6},   # Google Inc
    ],
    [
        {"label": "LOC", "start": 1, "end": 1},   # Paris
        {"label": "PER", "start": 2, "end": 3},   # Marie Curie
    ]
]

pred_spans = [
    [
        {"label": "PER", "start": 1, "end": 2},   # Correct
        {"label": "ORG", "start": 5, "end": 6},   # Correct
    ],
    [
        {"label": "LOC", "start": 1, "end": 2},   # Wrong boundary (extends into next entity)
        {"label": "PER", "start": 3, "end": 3},   # Partial match (missed start)
    ]
]

evaluator = Evaluator(
    true=true_spans,
    pred=pred_spans,
    tags=['PER', 'ORG', 'LOC'],
    loader="dict"
)

results = evaluator.evaluate()
print(evaluator.summary_report(mode='overall'))
```

--------------------------------

### Clone the repository

Source: https://github.com/mantisai/nervaluate/blob/main/CONTRIBUTING.md

Initial steps to set up the local development environment by cloning the fork.

```bash
git clone https://github.com/your-username/nervaluate.git
cd nervaluate
```

--------------------------------

### Run tests locally

Source: https://github.com/mantisai/nervaluate/blob/main/CONTRIBUTING.md

Command to execute the test suite using hatch.

```bash
hatch -e 
```

--------------------------------

### Initialize and Evaluate NER with Evaluator Class

Source: https://context7.com/mantisai/nervaluate/llms.txt

Demonstrates initializing the Evaluator with BIO-tagged lists for ground truth and predictions. Specifies entity tags and the loader type. Shows how to run the evaluation and access detailed results including precision, recall, and F1 scores for the 'strict' scenario.

```python
from nervaluate.evaluator import Evaluator

# Ground truth and predictions as BIO-tagged lists
# Each inner list represents one document/sentence
true = [
    ['O', 'B-PER', 'I-PER', 'O', 'O', 'O', 'B-ORG', 'I-ORG'],  # "The John Smith who works at Google Inc"
    ['O', 'B-LOC', 'B-PER', 'I-PER', 'O', 'O', 'B-DATE'],      # "In Paris Marie Curie lived in 1895"
]

pred = [
    ['O', 'O', 'B-PER', 'I-PER', 'O', 'O', 'B-ORG', 'I-ORG'],  # Missed start of PER entity
    ['O', 'B-LOC', 'I-LOC', 'B-PER', 'O', 'O', 'B-DATE'],      # LOC extends into PER
]

# Initialize evaluator with entity tags to evaluate
evaluator = Evaluator(
    true=true,
    pred=pred,
    tags=['PER', 'ORG', 'LOC', 'DATE'],
    loader="list",
    min_overlap_percentage=1.0  # Minimum overlap threshold (1-100%)
)

# Run evaluation and get detailed results
results = evaluator.evaluate()

# Results structure:
# {
#     "overall": {strategy_name: EvaluationResult, ...},
#     "entities": {entity_type: {strategy_name: EvaluationResult, ...}, ...},
#     "overall_indices": {strategy_name: EvaluationIndices, ...},
#     "entity_indices": {entity_type: {strategy_name: EvaluationIndices, ...}, ...}
# }

print(f"Strict Precision: {results['overall']['strict'].precision:.2f}")
print(f"Strict Recall: {results['overall']['strict'].recall:.2f}")
print(f"Strict F1: {results['overall']['strict'].f1:.2f}")
```

--------------------------------

### Create and Use EvaluationResult

Source: https://context7.com/mantisai/nervaluate/llms.txt

Illustrates the creation of an `EvaluationResult` object, which stores NER evaluation metrics. It also shows how to compute precision, recall, and F1 scores using the `compute_metrics` method.

```python
from nervaluate.entities import EvaluationResult

result = EvaluationResult(
    correct=8,
    incorrect=2,
    partial=1,
    missed=3,
    spurious=1
)
result.compute_metrics(partial_or_type=True)  # Use partial match formula
print(f"Precision: {result.precision:.2f}")
print(f"Recall: {result.recall:.2f}")
print(f"F1: {result.f1:.2f}")
print(f"Actual (TP+FP): {result.actual}")
print(f"Possible (TP+FN): {result.possible}")
```

--------------------------------

### Calculate Possible and Actual Quantities

Source: https://github.com/mantisai/nervaluate/blob/main/README.md

Formulas for determining the total possible and actual entities for metric calculations.

```text
POSSIBLE (POS) = COR + INC + PAR + MIS = TP + FN
ACTUAL (ACT) = COR + INC + PAR + SPU = TP + FP
```

--------------------------------

### Evaluate NER Performance with Strategies

Source: https://context7.com/mantisai/nervaluate/llms.txt

Assess NER performance using different evaluation strategies. The min_overlap_percentage parameter can be used to adjust boundary requirements.

```python
from nervaluate.evaluator import Evaluator

# Example with boundary errors
true = [['O', 'B-PER', 'I-PER', 'I-PER', 'O']]  # "John David Smith"
pred = [['O', 'O', 'B-PER', 'I-PER', 'O']]     # "David Smith" - missed first token

evaluator = Evaluator(true, pred, tags=['PER'], loader="list")
results = evaluator.evaluate()
```

--------------------------------

### Evaluation Strategies

Source: https://context7.com/mantisai/nervaluate/llms.txt

Details the different evaluation strategies and the `min_overlap_percentage` parameter.

```APIDOC
## Evaluation Strategies

### Description
Nervaluate implements four evaluation strategies that assess different aspects of NER performance. The `min_overlap_percentage` parameter controls how much overlap is required between predicted and true entity boundaries.

### Strategies
- **Strict**: Requires exact match of boundaries and entity type.
- **Partial**: Allows partial overlap if the entity type matches.
- **Entity Type**: Matches based on entity type only, ignoring boundaries.
- **Exact**: Matches based on exact boundary and entity type, similar to strict but may have nuances in implementation.

### `min_overlap_percentage` Parameter
This parameter, used within certain strategies (like 'partial'), defines the minimum required overlap percentage between predicted and true entity spans for them to be considered a match.

### Request Example
```python
from nervaluate.evaluator import Evaluator

# Example with boundary errors
true = [['O', 'B-PER', 'I-PER', 'I-PER', 'O']]  # "John David Smith"
pred = [['O', 'O', 'B-PER', 'I-PER', 'O']]     # "David Smith" - missed first token

evaluator = Evaluator(true, pred, tags=['PER'], loader="list")
results = evaluator.evaluate()
```
```

--------------------------------

### Track Entity Indices with EvaluationIndices

Source: https://context7.com/mantisai/nervaluate/llms.txt

Demonstrates the use of `EvaluationIndices` to keep track of the specific positions of correct and missed entities across documents. This provides granular insight into evaluation outcomes.

```python
from nervaluate.entities import EvaluationIndices

indices = EvaluationIndices()
indices.correct_indices.append((0, 1))  # Document 0, Entity 1
indices.missed_indices.append((0, 2))   # Document 0, Entity 2
print(f"Correct entities: {indices.correct_indices}")
print(f"Missed entities: {indices.missed_indices}")
```

--------------------------------

### Calculate Partial Match Metrics

Source: https://github.com/mantisai/nervaluate/blob/main/README.md

Precision and recall formulas for partial and type evaluation schemas, incorporating a 0.5 weight for partial matches.

```text
Precision = (COR + 0.5 × PAR) / ACT = TP / (TP + FP)
Recall = (COR + 0.5 × PAR)/POS = COR / ACT = TP / (TP + FN)
```

--------------------------------

### Entity Object Comparison

Source: https://context7.com/mantisai/nervaluate/llms.txt

Shows how `Entity` objects can be compared for equality. Two entities are considered equal if they have the same label and span.

```python
from nervaluate.entities import Entity

entity = Entity(label="PER", start=1, end=3)
entity2 = Entity(label="PER", start=1, end=3)
entity3 = Entity(label="PER", start=1, end=4)
print(f"Same entity: {entity == entity2}")  # True
print(f"Different entity: {entity == entity3}")  # False
```

--------------------------------

### Print Summary Report

Source: https://github.com/mantisai/nervaluate/blob/main/README.md

Generate and print evaluation reports for different scenarios or aggregated by entity type.

```python
print(evaluator.summary_report())
```

```python
print(evaluator.summary_report(mode='entities'))
```

--------------------------------

### Generate changelog

Source: https://github.com/mantisai/nervaluate/blob/main/CONTRIBUTING.md

Command to update the CHANGELOG.rst file based on commit history.

```bash
gitchangelog > CHANGELOG.rst
```

--------------------------------

### Initialize Evaluator

Source: https://github.com/mantisai/nervaluate/blob/main/README.md

Initialize the Evaluator class with true and predicted labels, specifying the entity types and loader format.

```python
from nervaluate.evaluator import Evaluator

true = [
    ['O', 'B-PER', 'I-PER', 'O', 'O', 'O', 'B-ORG', 'I-ORG'],  # "The John Smith who works at Google Inc"
    ['O', 'B-LOC', 'B-PER', 'I-PER', 'O', 'O', 'B-DATE'],      # "In Paris Marie Curie lived in 1895"
]
  
pred = [
    ['O', 'O', 'B-PER', 'I-PER', 'O', 'O', 'B-ORG', 'I-ORG'],
    ['O', 'B-LOC', 'I-LOC', 'B-PER', 'O', 'O', 'B-DATE'],
]
   
evaluator = Evaluator(true, pred, tags=['PER', 'ORG', 'LOC', 'DATE'], loader="list")
```

--------------------------------

### Calculate Exact Match Metrics

Source: https://github.com/mantisai/nervaluate/blob/main/README.md

Precision and recall formulas for strict and exact evaluation schemas.

```text
Precision = (COR / ACT) = TP / (TP + FP)
Recall = (COR / POS) = TP / (TP+FN)
```

--------------------------------

### Dictionary/Span Format Loader

Source: https://context7.com/mantisai/nervaluate/llms.txt

Accepts pre-computed entity spans as dictionaries for evaluation.

```APIDOC
## Dictionary/Span Format Loader

### Description
The `DictLoader` accepts pre-computed entity spans as dictionaries with `label`, `start`, and `end` keys, compatible with annotation tools like Prodigy.

### Usage
Initialize `Evaluator` with `loader="dict"` and provide lists of dictionaries representing entity spans for `true` and `pred`.

### Request Example
```python
from nervaluate.evaluator import Evaluator

# Dictionary format: list of documents, each containing list of entity spans
true_spans = [
    [
        {"label": "PER", "start": 1, "end": 2},   # John Smith
        {"label": "ORG", "start": 5, "end": 6},   # Google Inc
    ],
    [
        {"label": "LOC", "start": 1, "end": 1},   # Paris
        {"label": "PER", "start": 2, "end": 3},   # Marie Curie
    ]
]

pred_spans = [
    [
        {"label": "PER", "start": 1, "end": 2},   # Correct
        {"label": "ORG", "start": 5, "end": 6},   # Correct
    ],
    [
        {"label": "LOC", "start": 1, "end": 2},   # Wrong boundary (extends into next entity)
        {"label": "PER", "start": 3, "end": 3},   # Partial match (missed start)
    ]
]

evaluator = Evaluator(
    true=true_spans,
    pred=pred_spans,
    tags=['PER', 'ORG', 'LOC'],
    loader="dict"
)

results = evaluator.evaluate()
print(evaluator.summary_report(mode='overall'))
```
```

--------------------------------

### Create and Print an Entity Object

Source: https://context7.com/mantisai/nervaluate/llms.txt

Demonstrates the creation of an `Entity` object, which represents a single named entity with its label and character positions. This class is part of the data structures for NER results.

```python
from nervaluate.entities import Entity

entity = Entity(label="PER", start=1, end=3)
print(f"Entity: {entity.label} at positions {entity.start}-{entity.end}")
```

--------------------------------

### Commit message format

Source: https://github.com/mantisai/nervaluate/blob/main/CONTRIBUTING.md

Standard structure for commit messages to ensure consistency.

```text
type(scope): subject

body
```

--------------------------------

### CoNLL Format Loader

Source: https://context7.com/mantisai/nervaluate/llms.txt

Handles tab-separated CoNLL format data for evaluation.

```APIDOC
## CoNLL Format Loader

### Description
The `ConllLoader` handles tab-separated CoNLL format data where each line contains a token and its BIO tag, with documents separated by blank lines.

### Usage
Initialize `Evaluator` with `loader="conll"` and provide CoNLL formatted strings for `true` and `pred`.

### Request Example
```python
from nervaluate.evaluator import Evaluator

# CoNLL format: token<TAB>tag, documents separated by blank lines
true_conll = """
The\tO
John\tB-PER
Smith\tI-PER
works\tO
at\tO
Google\tB-ORG
Inc\tI-ORG

In\tO
Paris\tB-LOC
she\tO
lived\tO"""

pred_conll = """
The\tO
John\tB-PER
Smith\tI-PER
works\tO
at\tO
Google\tB-ORG
Inc\tI-ORG

In\tO
Paris\tB-LOC
Marie\tB-PER
lived\tO"""

evaluator = Evaluator(
    true=true_conll,
    pred=pred_conll,
    tags=['PER', 'ORG', 'LOC'],
    loader="conll"
)

results = evaluator.evaluate()
print(f"Document 1 - Strict F1: {results['overall']['strict'].f1:.2f}")
```

### Response Example
```
Document 1 - Strict F1: 0.75
```
```

--------------------------------

### Print Overall NER Evaluation Summary

Source: https://context7.com/mantisai/nervaluate/llms.txt

Prints a formatted overall summary of the NER evaluation results. This is useful for a quick overview of the model's performance across all entities and scenarios.

```python
print("=" * 60)
print("OVERALL EVALUATION RESULTS")
print("=" * 60)
print(evaluator.summary_report(mode='overall'))
```

--------------------------------

### NER Evaluation Modes

Source: https://context7.com/mantisai/nervaluate/llms.txt

Demonstrates different evaluation modes for NER results, including strict, exact, partial, and entity type matching. Use these to control how boundary and type matches are scored.

```python
strict = results['overall']['strict']
print(f"Strict - Correct: {strict.correct}, Incorrect: {strict.incorrect}")
```

```python
exact = results['overall']['exact']
print(f"Exact - Correct: {exact.correct}, Incorrect: {exact.incorrect}")
```

```python
partial = results['overall']['partial']
print(f"Partial - Correct: {partial.correct}, Partial: {partial.partial}")
```

```python
ent_type = results['overall']['ent_type']
print(f"Ent_Type - Correct: {ent_type.correct}, Incorrect: {ent_type.incorrect}")
```

--------------------------------

### Direct NER Evaluation with List Loader

Source: https://context7.com/mantisai/nervaluate/llms.txt

Use this method for direct evaluation when your true and predicted entities are provided as lists. Ensure 'y_true' and 'y_pred' are correctly formatted lists of entities. The 'tags' parameter specifies the entity types to evaluate.

```python
entity_tags = ['PER', 'ORG', 'LOC', 'DATE']
evaluator = Evaluator(y_true, y_pred, tags=entity_tags, loader="list")
results = evaluator.evaluate()
```

--------------------------------

### NER Evaluation with Span Dictionary Loader

Source: https://context7.com/mantisai/nervaluate/llms.txt

Use this method when your true and predicted entities are pre-converted into span dictionaries. This is particularly useful when working with custom NER models that output spans. It allows for direct comparison with list-based evaluation.

```python
true_spans = [collect_named_entities(doc) for doc in y_true]
pred_spans = [collect_named_entities(doc) for doc in y_pred]

evaluator_spans = Evaluator(true_spans, pred_spans, tags=entity_tags, loader="dict")
print("\nResults from span-based input match:",
      evaluator.evaluate()['overall']['strict'].f1 == evaluator_spans.evaluate()['overall']['strict'].f1)
```

--------------------------------

### Load CoNLL Format Data

Source: https://context7.com/mantisai/nervaluate/llms.txt

Use the ConllLoader to process tab-separated BIO tag data. Documents must be separated by blank lines.

```python
from nervaluate.evaluator import Evaluator

# CoNLL format: token<TAB>tag, documents separated by blank lines
true_conll = """The	O
John	B-PER
Smith	I-PER
works	O
at	O
Google	B-ORG
Inc	I-ORG

In	O
Paris	B-LOC
she	O
lived	O"""

pred_conll = """The	O
John	B-PER
Smith	I-PER
works	O
at	O
Google	B-ORG
Inc	I-ORG

In	O
Paris	B-LOC
Marie	B-PER
lived	O"""

evaluator = Evaluator(
    true=true_conll,
    pred=pred_conll,
    tags=['PER', 'ORG', 'LOC'],
    loader="conll"
)

results = evaluator.evaluate()
print(f"Document 1 - Strict F1: {results['overall']['strict'].f1:.2f}")
# Output: Document 1 - Strict F1: 0.75
```

--------------------------------

### Export Evaluation Results to CSV

Source: https://context7.com/mantisai/nervaluate/llms.txt

Export metrics to a CSV string or file using results_to_csv. Supports different modes like 'overall' or 'entities'.

```python
from nervaluate.evaluator import Evaluator

true = [['O', 'B-PER', 'I-PER', 'O', 'B-ORG', 'I-ORG']]
pred = [['O', 'B-PER', 'I-PER', 'O', 'B-ORG', 'I-ORG']]

evaluator = Evaluator(true, pred, tags=['PER', 'ORG'], loader="list")

# Export overall results as CSV string
csv_string = evaluator.results_to_csv(mode='overall')
print(csv_string)
# Output:
# Strategy,Correct,Incorrect,Partial,Missed,Spurious,Precision,Recall,F1-Score
# strict,2,0,0,0,0,1.0,1.0,1.0
# partial,2,0,0,0,0,1.0,1.0,1.0
# ent_type,2,0,0,0,0,1.0,1.0,1.0
# exact,2,0,0,0,0,1.0,1.0,1.0

# Export per-entity results for strict scenario to file
evaluator.results_to_csv(mode='entities', scenario='strict', file_path='results.csv')

# Export per-entity results as string
csv_entities = evaluator.results_to_csv(mode='entities', scenario='partial')
print(csv_entities)
# Output:
# Entity,Correct,Incorrect,Partial,Missed,Spurious,Precision,Recall,F1-Score
# PER,1,0,0,0,0,1.0,1.0,1.0
# ORG,1,0,0,0,0,1.0,1.0,1.0
```

--------------------------------

### Print Per-Entity NER Evaluation Breakdown

Source: https://context7.com/mantisai/nervaluate/llms.txt

Prints a detailed breakdown of NER evaluation results per entity type for a specified scenario. Use this to analyze performance on specific entity types and understand different error types (strict, exact, partial, ent_type).

```python
for scenario in ['strict', 'exact', 'partial', 'ent_type']:
    print(f"\n{'=' * 60}")
    print(f"PER-ENTITY RESULTS - {scenario.upper()}")
    print("=" * 60)
    print(evaluator.summary_report(mode='entities', scenario=scenario))
```

--------------------------------

### Results to CSV - Export Metrics

Source: https://context7.com/mantisai/nervaluate/llms.txt

Exports evaluation results to CSV format, either saving to a file or returning as a string.

```APIDOC
## `results_to_csv()`

### Description
Exports evaluation results to CSV format, either saving to a file or returning as a string for further processing.

### Method
`results_to_csv(mode='overall', scenario='strict', file_path=None)`

### Parameters
#### Query Parameters
- **mode** (string) - Optional - Specifies the reporting mode. Can be 'overall' or 'entities'. Defaults to 'overall'.
- **scenario** (string) - Optional - The evaluation scenario to export. Defaults to 'strict'.
- **file_path** (string) - Optional - The path to save the CSV file. If None, the CSV content is returned as a string.

### Request Example
```python
from nervaluate.evaluator import Evaluator

true = [['O', 'B-PER', 'I-PER', 'O', 'B-ORG', 'I-ORG']]
pred = [['O', 'B-PER', 'I-PER', 'O', 'B-ORG', 'I-ORG']]

evaluator = Evaluator(true, pred, tags=['PER', 'ORG'], loader="list")

# Export overall results as CSV string
csv_string = evaluator.results_to_csv(mode='overall')
print(csv_string)

# Export per-entity results for strict scenario to file
evaluator.results_to_csv(mode='entities', scenario='strict', file_path='results.csv')

# Export per-entity results as string
csv_entities = evaluator.results_to_csv(mode='entities', scenario='partial')
print(csv_entities)
```

### Response
#### Success Response (200)
- **CSV Content** (string) - If `file_path` is None, returns the evaluation results as a CSV formatted string.
- **File Saved** - If `file_path` is provided, the results are saved to the specified file.

#### Response Example (CSV String)
```
Strategy,Correct,Incorrect,Partial,Missed,Spurious,Precision,Recall,F1-Score
strict,2,0,0,0,0,1.0,1.0,1.0
partial,2,0,0,0,0,1.0,1.0,1.0
ent_type,2,0,0,0,0,1.0,1.0,1.0
exact,2,0,0,0,0,1.0,1.0,1.0
```
```

--------------------------------

### Track Entity Classification Errors with summary_report_indices

Source: https://context7.com/mantisai/nervaluate/llms.txt

Use this method to identify specific entities classified as correct, incorrect, partial, missed, or spurious. Requires an initialized Evaluator object.

```python
from nervaluate.evaluator import Evaluator

true = [
    ['O', 'B-PER', 'I-PER', 'O', 'B-ORG'],
    ['O', 'B-LOC', 'O'],
]

pred = [
    ['O', 'B-PER', 'I-PER', 'O', 'B-ORG'],  # Perfect match
    ['O', 'O', 'B-LOC'],                     # LOC boundary wrong
]

evaluator = Evaluator(true, pred, tags=['PER', 'ORG', 'LOC'], loader="list")

# Get detailed indices report for strict evaluation
print(evaluator.summary_report_indices(mode='overall', scenario='strict', colors=False))
# Output:
# Indices for error schema 'strict':
#
# Category             Instance   Entity   Details
# ------------------------------------------------------------------------------
# Correct              0          0        Label=PER, Start=1, End=2
# Correct              0          1        Label=ORG, Start=4, End=4
# Incorrect            1          0        Label=LOC, Start=2, End=2
# Missed               1          0        Label=LOC, Start=1, End=1

# Per-entity indices report
print(evaluator.summary_report_indices(mode='entities', scenario='strict'))
```

--------------------------------

### Custom Overlap Threshold for Evaluation

Source: https://context7.com/mantisai/nervaluate/llms.txt

Configures the Evaluator with a custom minimum overlap percentage. This is useful when partial matches require a significant portion of the entity to overlap.

```python
evaluator_50 = Evaluator(true, pred, tags=['PER'], loader="list", min_overlap_percentage=50.0)
results_50 = evaluator_50.evaluate()
print(f"With 50% threshold - Partial correct: {results_50['overall']['partial'].partial}")
```

--------------------------------

### Collect Named Entities from BIO Tags

Source: https://context7.com/mantisai/nervaluate/llms.txt

Converts a list of BIO-tagged tokens into a list of dictionaries, where each dictionary represents a named entity with its label and span. This is useful for processing raw model outputs.

```python
from nervaluate.utils import collect_named_entities

bio_tags = ['O', 'B-PER', 'I-PER', 'O', 'B-ORG', 'I-ORG', 'I-ORG', 'O', 'B-LOC']
entities = collect_named_entities(bio_tags)
print(entities)
```

--------------------------------

### Export NER Evaluation Results to CSV

Source: https://context7.com/mantisai/nervaluate/llms.txt

Exports the overall NER evaluation results into a CSV formatted string. This is useful for external analysis, reporting, or integration with other data processing tools.

```python
csv_output = evaluator.results_to_csv(mode='overall')
print("\nCSV Export:")
print(csv_output)
```

--------------------------------

### Convert Multiple Documents to Spans

Source: https://context7.com/mantisai/nervaluate/llms.txt

Processes a list of documents, where each document is a list of BIO tags, and converts them into a list of span dictionaries. This function is part of the utility functions for data preprocessing.

```python
from nervaluate.utils import list_to_spans

documents = [
    ['O', 'B-PER', 'I-PER', 'O'],
    ['B-LOC', 'O', 'B-ORG', 'I-ORG'],
]
spans = list_to_spans(documents)
print(spans)
```

--------------------------------

### Summary Report Indices - Error Analysis

Source: https://context7.com/mantisai/nervaluate/llms.txt

Provides detailed tracking of entities classified as correct, incorrect, partial, missed, or spurious for fine-grained error analysis.

```APIDOC
## `summary_report_indices()`

### Description
Provides detailed tracking of entities classified as correct, incorrect, partial, missed, or spurious, enabling fine-grained error analysis.

### Method
`summary_report_indices(mode='overall', scenario='strict', colors=False)`

### Parameters
#### Query Parameters
- **mode** (string) - Optional - Specifies the reporting mode. Can be 'overall' or 'entities'.
- **scenario** (string) - Optional - The evaluation scenario to report on (e.g., 'strict', 'partial', 'ent_type', 'exact'). Defaults to 'strict'.
- **colors** (boolean) - Optional - Whether to use colors in the output. Defaults to False.

### Request Example
```python
from nervaluate.evaluator import Evaluator

true = [
    ['O', 'B-PER', 'I-PER', 'O', 'B-ORG'],
    ['O', 'B-LOC', 'O'],
]

pred = [
    ['O', 'B-PER', 'I-PER', 'O', 'B-ORG'],  # Perfect match
    ['O', 'O', 'B-LOC'],                     # LOC boundary wrong
]

evaluator = Evaluator(true, pred, tags=['PER', 'ORG', 'LOC'], loader="list")

# Get detailed indices report for strict evaluation
print(evaluator.summary_report_indices(mode='overall', scenario='strict', colors=False))

# Per-entity indices report
print(evaluator.summary_report_indices(mode='entities', scenario='strict'))
```

### Response
#### Success Response (200)
- **Indices Report** (string) - A formatted string detailing the error analysis results.

#### Response Example
```
Indices for error schema 'strict':

Category             Instance   Entity   Details
------------------------------------------------------------------------------
Correct              0          0        Label=PER, Start=1, End=2
Correct              0          1        Label=ORG, Start=4, End=4
Incorrect            1          0        Label=LOC, Start=2, End=2
Missed               1          0        Label=LOC, Start=1, End=1
```
```

--------------------------------

### Convert CoNLL Format to Spans

Source: https://context7.com/mantisai/nervaluate/llms.txt

Parses a string in CoNLL format, where each line contains a token and its BIO tag, and converts it into a list of span dictionaries. This utility is helpful for loading data from standard NER formats.

```python
from nervaluate.utils import conll_to_spans

conll_data = """John\tB-PER
Smith\tI-PER
works\tO

Paris\tB-LOC
is\tO"""
spans = conll_to_spans(conll_data)
print(spans)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.