### CLI: Visualize Similarity Matrix with bert-score-show

Source: https://context7.com/tiiiger/bert_score/llms.txt

Demonstrates the command-line tool `bert-score-show` for generating similarity matrix visualizations. Examples include basic usage, applying baseline rescaling, and specifying a custom model and layer count.

```bash
# Basic visualization
bert-score-show \
    --lang en \
    -r "There are two bananas on the table."
    -c "On the table are two apples."
    -f similarity_output.png

# With rescaling for clearer visualization
bert-score-show \
    --lang en \
    -r "The quick brown fox jumps over the lazy dog."
    -c "A fast brown fox leaps over a sleeping dog."
    -f fox_similarity.png \
    --rescale_with_baseline

# Using specific model
bert-score-show \
    -r "Machine learning enables pattern recognition."
    -c "ML allows recognizing patterns in data."
    -m microsoft/deberta-base \
    -l 9 \
    -f ml_similarity.png
```

--------------------------------

### Visualize Token Similarity Matrix with plot_example()

Source: https://context7.com/tiiiger/bert_score/llms.txt

Provides examples of using the `plot_example` function to visualize the token-level similarity matrix between candidate and reference sentences. It demonstrates saving the plot to a file and using baseline rescaling for clearer visualization. It also shows how to use the `BERTScorer` class's `plot_example` method.

```python
from bert_score import plot_example, BERTScorer

# Visualize similarity matrix between candidate and reference
candidate = "On the table are two apples."
reference = "There are two bananas on the table."

# Generate and display plot
plot_example(
    candidate,
    reference,
    lang="en",
    fname="similarity_matrix.png"  # Save to file
)

# With baseline rescaling for clearer visualization
plot_example(
    candidate,
    reference,
    lang="en",
    rescale_with_baseline=True,
    fname="similarity_matrix_rescaled.png"
)

# Using BERTScorer class for visualization
scorer = BERTScorer(lang="en", rescale_with_baseline=True)
scorer.plot_example(
    "The cat is sleeping on the couch.",
    "A cat sleeps on the sofa.",
    fname="cat_similarity.png"
)
```

--------------------------------

### Visualizing BERTScore Example Matchings

Source: https://github.com/tiiiger/bert_score/blob/master/example/Demo.ipynb

Utilizes the `plot_example` function from BERTScore to visualize the word-level matchings between a candidate sentence and a reference sentence. This helps in understanding how the scores are derived.

```python
from bert_score import plot_example

# Assuming cand and ref are single strings
# plot_example(cands[0], refs[0], lang="en")
```

--------------------------------

### Rescale BERTScore with Baseline via Command Line

Source: https://github.com/tiiiger/bert_score/blob/master/journal/rescale_baseline.md

This command-line example shows how to invoke the BERTScore tool with baseline rescaling. It takes paths to reference and candidate sentence files, specifies the language, and enables the `--rescale_with_baseline` flag.

```bash
bert-score -r example/refs.txt -c example/hyps.txt --lang en --rescale_with_baseline
```

--------------------------------

### Verify BERTScore Installation

Source: https://github.com/tiiiger/bert_score/blob/master/example/Demo.ipynb

This snippet imports the bert_score module and prints the installed version to ensure the package is correctly configured.

```python
import bert_score
print(bert_score.__version__)
```

--------------------------------

### Initialize BERTScorer with IDF Weighting and Custom Corpus

Source: https://context7.com/tiiiger/bert_score/llms.txt

Demonstrates initializing BERTScorer with IDF weighting using a custom corpus for calculating weights. It then scores a sentence pair and prints the F1 score. Also shows how to get the model hash for reproducibility.

```python
from bert_score import BERTScorer

idf_corpus = [
    "Natural language processing enables computers to understand text.",
    "Machine learning models require large amounts of training data.",
    "Deep learning has revolutionized artificial intelligence.",
    "Neural networks can learn complex patterns from data."
]

scorer_idf = BERTScorer(
    lang="en",
    idf=True,
    idf_sents=idf_corpus,  # Compute IDF weights from this corpus
    rescale_with_baseline=True
)

P, R, F1 = scorer_idf.score(
    ["Neural nets learn patterns."],
    ["Deep learning models recognize patterns."]
)
print(f"IDF-weighted F1: {F1.item():.4f}")

# Get the hash code for reproducibility
print(f"Model hash: {scorer_idf.hash}")
```

--------------------------------

### Visualizing BERTScore Example with Baseline Rescaling

Source: https://github.com/tiiiger/bert_score/blob/master/example/Demo.ipynb

Visualizes the word-level matchings for BERTScore, applying baseline rescaling to the similarity scores. This provides a more spread-out and interpretable visualization of the alignments.

```python
from bert_score import plot_example

# Assuming cand and ref are single strings
# plot_example(cands[0], refs[0], lang="en", rescale_with_baseline=True)
```

--------------------------------

### Install BERTScore via pip

Source: https://github.com/tiiiger/bert_score/blob/master/example/Demo.ipynb

This command installs the BERTScore library using the pip package manager. It is intended to be run in a Jupyter notebook environment.

```python
!pip install bert_score
```

--------------------------------

### Evaluating Machine Translation Output

Source: https://context7.com/tiiiger/bert_score/llms.txt

An example demonstrating how to use BERTScore to evaluate machine translation outputs against reference translations, including initialization of the `BERTScorer` with specific configurations and reporting of results.

```APIDOC
## Evaluating Machine Translation Output

### Description
This example illustrates how to leverage BERTScore for evaluating the quality of machine translation (MT) outputs. It initializes a `BERTScorer` with specific parameters like language, baseline rescaling, and model type, then computes precision, recall, and F1 scores against reference translations.

### Method
```python
from bert_score import BERTScorer
import json

# Initialize BERTScorer for translation evaluation
scorer = BERTScorer(
    lang="en",
    rescale_with_baseline=True,
    model_type="microsoft/deberta-large-mnli"
)

# Sample machine translation outputs and their references
mt_outputs = [
    "The meeting will be held tomorrow at 3 PM.",
    "Please send me the report by Friday.",
    "The project deadline has been extended."
]

reference_translations = [
    "The meeting is scheduled for tomorrow at 3 o'clock.",
    "Please email the report to me before Friday.",
    "The deadline for the project has been pushed back."
]

# Compute BERTScore metrics
P, R, F1 = scorer.score(mt_outputs, reference_translations, verbose=True)

# Format and print the results
results = {
    "model": scorer.hash, # Unique hash for reproducibility
    "avg_precision": P.mean().item(),
    "avg_recall": R.mean().item(),
    "avg_f1": F1.mean().item(),
    "per_sentence": [
        {"precision": p.item(), "recall": r.item(), "f1": f.item()} 
        for p, r, f in zip(P, R, F1)
    ]
}

print(json.dumps(results, indent=2))
```

### Parameters
- **lang** (str) - Language of the text (e.g., 'en').
- **rescale_with_baseline** (bool) - Whether to rescale scores with baseline values.
- **model_type** (str) - The pre-trained model to use (e.g., 'microsoft/deberta-large-mnli').
- **mt_outputs** (list of str) - List of machine-translated sentences.
- **reference_translations** (list of str) - List of reference sentences.
- **verbose** (bool, optional) - If True, prints progress messages.

### Response
- **model** (str) - Hash code of the scorer configuration for reproducibility.
- **avg_precision** (float) - Average precision score across all sentences.
- **avg_recall** (float) - Average recall score across all sentences.
- **avg_f1** (float) - Average F1 score across all sentences.
- **per_sentence** (list of dict) - A list containing precision, recall, and F1 scores for each sentence pair.
```

--------------------------------

### Batch Processing with Progress Tracking

Source: https://context7.com/tiiiger/bert_score/llms.txt

Illustrates how to efficiently evaluate large datasets by processing them in batches using `BERTScorer` and tracking progress with `tqdm`.

```APIDOC
## Batch Processing with Progress Tracking

### Description
This example demonstrates a robust method for evaluating large datasets with BERTScore by processing them in manageable batches. It utilizes the `tqdm` library to provide a progress bar, making it easier to monitor the evaluation of extensive text corpora.

### Method
```python
from bert_score import BERTScorer
from tqdm import tqdm
import torch

def evaluate_in_batches(scorer, candidates, references, batch_size=100):
    """Evaluate large datasets in batches."""
    all_P, all_R, all_F1 = [], [], []

    for i in tqdm(range(0, len(candidates), batch_size), desc="Evaluating Batches"):
        batch_cands = candidates[i:i+batch_size]
        batch_refs = references[i:i+batch_size]

        P, R, F1 = scorer.score(batch_cands, batch_refs)
        all_P.append(P)
        all_R.append(R)
        all_F1.append(F1)

    return (
        torch.cat(all_P),
        torch.cat(all_R),
        torch.cat(all_F1)
    )

# Usage example
scorer = BERTScorer(lang="en", rescale_with_baseline=True)

# Simulate a large dataset
candidates = ["sentence " + str(i) for i in range(1000)]
references = ["reference " + str(i) for i in range(1000)]

# Evaluate in batches
P, R, F1 = evaluate_in_batches(scorer, candidates, references, batch_size=64)

print(f"Evaluated {len(F1)} pairs, avg F1: {F1.mean().item():.4f}")
```

### Parameters
- **scorer** (BERTScorer) - An initialized BERTScorer object.
- **candidates** (list of str) - List of candidate sentences.
- **references** (list of str) - List of reference sentences.
- **batch_size** (int, optional) - The number of sentences to process in each batch. Defaults to 100.

### Response
- **all_P** (torch.Tensor) - Concatenated precision scores for all batches.
- **all_R** (torch.Tensor) - Concatenated recall scores for all batches.
- **all_F1** (torch.Tensor) - Concatenated F1 scores for all batches.
```

--------------------------------

### Implement Batch Processing with Progress Tracking

Source: https://context7.com/tiiiger/bert_score/llms.txt

Provides a function to process large datasets in batches using tqdm for progress tracking, which is essential for memory management during large-scale evaluations.

```python
from bert_score import BERTScorer
from tqdm import tqdm
import torch

def evaluate_in_batches(scorer, candidates, references, batch_size=100):
    all_P, all_R, all_F1 = [], [], []
    for i in tqdm(range(0, len(candidates), batch_size)):
        batch_cands = candidates[i:i+batch_size]
        batch_refs = references[i:i+batch_size]
        P, R, F1 = scorer.score(batch_cands, batch_refs)
        all_P.append(P)
        all_R.append(R)
        all_F1.append(F1)
    return torch.cat(all_P), torch.cat(all_R), torch.cat(all_F1)
```

--------------------------------

### Download WMT16 Dataset using Bash

Source: https://github.com/tiiiger/bert_score/blob/master/tune_layers/README.md

This script downloads the WMT16 dataset and extracts it into a 'wmt16' folder. It checks if the folder already exists to avoid redundant downloads.

```bash
bash download_data.sh
```

--------------------------------

### Utility: Compute IDF Weights with get_idf_dict()

Source: https://context7.com/tiiiger/bert_score/llms.txt

Shows how to use the `get_idf_dict` utility function to compute inverse document frequency (IDF) weights for tokens within a given corpus. It also demonstrates obtaining a tokenizer, which is often a prerequisite for IDF calculations.

```python
from bert_score.utils import get_idf_dict, get_tokenizer

# Get tokenizer for the model
tokenizer = get_tokenizer("roberta-large")

# Example usage (assuming idf_corpus is defined elsewhere)
# idf_dict = get_idf_dict(idf_corpus, tokenizer)
# print(idf_dict)
```

--------------------------------

### Tune Models for Rescale Baselines using Python

Source: https://github.com/tiiiger/bert_score/blob/master/get_rescale_baseline/README.md

This Python script tunes models to obtain rescale baseline files. It requires specifying the language and model names. The output files are stored in the 'rescale_baseline' folder.

```python
python get_rescale_baseline.py --lang en -b 16 -m \
    microsoft/deberta-large \
    microsoft/deberta-large-mnli
```

--------------------------------

### BERTScore with Multiple Reference Files

Source: https://github.com/tiiiger/bert_score/blob/master/README.md

This command demonstrates how to use BERTScore with multiple reference files. The `-r` argument accepts an arbitrary number of reference files, each corresponding line-by-line to the candidate file. This is useful when evaluating against several reference texts.

```bash
bert-score -r example/refs.txt example/refs2.txt -c example/hyps.txt --lang en
```

--------------------------------

### CLI: Calculate BERTScore from Files with bert-score

Source: https://context7.com/tiiiger/bert_score/llms.txt

Illustrates the command-line interface for calculating BERTScore between text files. Covers basic usage, baseline rescaling, multiple references, specific model selection, IDF weighting, and evaluation for different languages like Chinese. Also shows options for controlling batch size and verbosity.

```bash
# Basic usage - evaluate English text files
bert-score -r refs.txt -c hyps.txt --lang en

# Output: roberta-large_L17_no-idf_version=0.3.12(hug_trans=4.x.x) P: 0.957378 R: 0.961325 F1: 0.959333

# With baseline rescaling for more interpretable scores
bert-score -r refs.txt -c hyps.txt --lang en --rescale_with_baseline

# Output: roberta-large_L17_no-idf_version=0.3.12(hug_trans=4.x.x)-rescaled P: 0.747044 R: 0.770484 F1: 0.759045

# Show individual scores for each sentence pair
bert-score -r refs.txt -c hyps.txt --lang en -s

# Multiple reference files (each line i in all ref files corresponds to candidate line i)
bert-score -r refs1.txt refs2.txt refs3.txt -c hyps.txt --lang en

# Using a specific model
bert-score -r refs.txt -c hyps.txt -m microsoft/deberta-xlarge-mnli -l 40

# With IDF weighting
bert-score -r refs.txt -c hyps.txt --lang en --idf

# Chinese text evaluation
bert-score -r chinese_refs.txt -c chinese_hyps.txt --lang zh

# Custom model with specific layer count
bert-score -r refs.txt -c hyps.txt -m path/to/my/bert --num_layers 9

# Control batch size for memory management
bert-score -r refs.txt -c hyps.txt --lang en -b 32 --verbose

# Full example with all options
bert-score \
    -r refs.txt \
    -c hyps.txt \
    --lang en \
    --rescale_with_baseline \
    --idf \
    -m microsoft/deberta-large-mnli \
    -l 18 \
    -b 64 \
    --nthreads 4 \
    -s \
    -v
```

--------------------------------

### Visualize BERTScore Matching with CLI

Source: https://github.com/tiiiger/bert_score/blob/master/README.md

This command generates a visualization of matching scores between a reference and a candidate sentence. The output is saved to a specified file (e.g., `out.png`). It requires the language, reference sentence, candidate sentence, and an output file path.

```bash
bert-score-show --lang en -r "There are two bananas on the table." -c "On the table are two apples." -f out.png
```

--------------------------------

### Access Language and Model Configurations

Source: https://context7.com/tiiiger/bert_score/llms.txt

Retrieves default model mappings for specific languages and identifies the optimal layer configurations for various transformer models to ensure high-quality evaluation.

```python
from bert_score.utils import lang2model, model2layers

# Default models for each language
print(lang2model["en"])      # roberta-large
print(lang2model["zh"])      # bert-base-chinese

# Optimal layers for each model
print(model2layers["roberta-large"])                    # 17
print(model2layers["microsoft/deberta-xlarge-mnli"])    # 40

# List all supported models
print(f"Supported models: {len(model2layers)}")
```

--------------------------------

### Download WMT17 English Text Data using Bash

Source: https://github.com/tiiiger/bert_score/blob/master/get_rescale_baseline/README.md

This script downloads the WMT17 English text data. It is a prerequisite for further model tuning.

```bash
bash download_text_data.sh
```

--------------------------------

### Evaluate Multiple Batches with BERTScorer Class

Source: https://context7.com/tiiiger/bert_score/llms.txt

Shows how to instantiate the BERTScorer class to cache model weights, which is more efficient for evaluating multiple batches of text. This approach avoids the overhead of reloading the model for each call.

```python
from bert_score import BERTScorer

# Initialize scorer with model caching
scorer = BERTScorer(lang="en", rescale_with_baseline=True, device="cuda")

# Score multiple batches
batch1_cands = ["The quick brown fox jumps.", "Hello world program."]
batch1_refs = ["A fast brown fox leaps.", "Hello world application."]
P1, R1, F1_1 = scorer.score(batch1_cands, batch1_refs, verbose=True)

batch2_cands = ["Machine learning is powerful.", "Deep neural networks learn."]
batch2_refs = ["ML techniques are very powerful.", "Deep learning models train."]
P2, R2, F1_2 = scorer.score(batch2_cands, batch2_refs)
```

--------------------------------

### Load Custom Model with BERTScore CLI

Source: https://github.com/tiiiger/bert_score/blob/master/README.md

This command shows how to use a custom BERT model with BERTScore. Specify the path to your model using `--model` and the number of layers to use with `--num_layers`. This allows for evaluation with models not included in the default distribution.

```bash
bert-score -r example/refs.txt -c example/hyps.txt --model path_to_my_bert --num_layers 9
```