### Install SacreBLEU via pip

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Standard installation commands for the SacreBLEU Python package, including optional dependencies for specific language tokenizers.

```bash
pip install sacrebleu
pip install "sacrebleu[ja]"
pip install "sacrebleu[ko]"
```

--------------------------------

### Run Paired Approximate Randomization Test with SacreBLEU CLI

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

This command-line example demonstrates how to perform a paired approximate randomization test using SacreBLEU. It compares multiple system outputs against a baseline using BLEU and chrF metrics, providing p-values to assess statistical significance.

```bash
$ sacrebleu -t wmt17 -l en-de -i newstest2017.LIUM-NMT.4900.en-de newstest2017.online-* -m bleu chrf --paired-ar
```

--------------------------------

### Scoring Machine Translation Outputs with SacreBLEU

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Examples of scoring machine translation outputs using SacreBLEU. It covers redirecting system output to STDIN and using the --input/-i argument. Options for short signatures, score-only output, and configuring score precision are also shown.

```bash
# Option 1: Redirect system output to STDIN
$ cat output.detok.txt | sacrebleu -t wmt17 -l en-de
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 20.8 54.4/26.6/14.9/8.7 (BP = 1.000 ratio = 1.026 hyp_len = 62880 ref_len = 61287)
```

```bash
# Option 2: Use the --input/-i argument
$ sacrebleu -t wmt17 -l en-de -i output.detok.txt
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 20.8 54.4/26.6/14.9/8.7 (BP = 1.000 ratio = 1.026 hyp_len = 62880 ref_len = 61287)
```

```bash
$ sacrebleu -t wmt17 -l en-de -i output.detok.txt -sh
BLEU|#:1|c:mixed|e:no|tok:13a|s:exp|v:2.0.0 = 20.8 54.4/26.6/14.9/8.7 (BP = 1.000 ratio = 1.026 hyp_len = 62880 ref_len = 61287)
```

```bash
$ sacrebleu -t wmt17 -l en-de -i output.detok.txt -b
20.8
```

```bash
$ sacrebleu -t wmt17 -l en-de -i output.detok.txt -b -w 4
20.7965
```

--------------------------------

### Download SacreBLEU Test Sets

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Commands to download and use SacreBLEU test sets. It shows how to download source files, pass them through a translation system, and score them. It also demonstrates how to access system outputs for specific test sets.

```bash
$ sacrebleu -t wmt17 -l en-de --echo src > wmt17.en-de.en
$ cat wmt17.en-de.en | translate.sh | sacrebleu -t wmt17 -l en-de
```

```bash
$ sacrebleu -t wmt21/systems -l zh-en --echo NiuTrans
```

```bash
$ sacrebleu -t wmt21/system -l zh-en --echo NiuTrans | sacrebleu -t wmt21/systems -l zh-en
```

--------------------------------

### Display SacreBLEU Help for Metric-Specific Arguments

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

This demonstrates how to view detailed arguments for specific metrics like BLEU, chrF, and TER. The output lists available options for customization.

```bash
BLEU related arguments:
  --smooth-method {none,floor,add-k,exp}, -s {none,floor,add-k,exp}
                        Smoothing method: exponential decay, floor (increment zero counts), add-k (increment num/denom by k for n>1), or none. (Default: exp)
  --smooth-value BLEU_SMOOTH_VALUE, -sv BLEU_SMOOTH_VALUE
                        The smoothing value. Only valid for floor and add-k. (Defaults: floor: 0.1, add-k: 1)
  --tokenize {none,zh,13a,char,intl,ja-mecab,ko-mecab}, -tok {none,zh,13a,char,intl,ja-mecab,ko-mecab}
                        Tokenization method to use for BLEU. If not provided, defaults to `zh` for Chinese, `ja-mecab` for Japanese, `ko-mecab` for Korean and `13a` (mteval) otherwise.
  --lowercase, -lc      If True, enables case-insensitivity. (Default: False)
  --force               Insist that your tokenized input is actually detokenized.

chrF related arguments:
  --chrf-char-order CHRF_CHAR_ORDER, -cc CHRF_CHAR_ORDER
                        Character n-gram order. (Default: 6)
  --chrf-word-order CHRF_WORD_ORDER, -cw CHRF_WORD_ORDER
                        Word n-gram order (Default: 0). If equals to 2, the metric is referred to as chrF++.
  --chrf-beta CHRF_BETA
                        Determine the importance of recall w.r.t precision. (Default: 2)
  --chrf-whitespace     Include whitespaces when extracting character n-grams. (Default: False)
  --chrf-lowercase      Enable case-insensitivity. (Default: False)
  --chrf-eps-smoothing  Enables epsilon smoothing similar to chrF++.py, NLTK and Moses; instead of effective order smoothing. (Default: False)

TER related arguments (The defaults replicate TERCOM's behavior):
  --ter-case-sensitive  Enables case sensitivity (Default: False)
  --ter-asian-support   Enables special treatment of Asian characters (Default: False)
  --ter-no-punct        Removes punctuation. (Default: False)
  --ter-normalized      Applies basic normalization and tokenization. (Default: False)
```

--------------------------------

### Perform Multi-reference Evaluation

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Shows how to evaluate a system output against multiple reference files using positional arguments or a tab-separated file.

```bash
# Using multiple references as positional arguments
sacrebleu ref1 ref2 -i system -m bleu chrf ter

# Using a tab-separated file with --num-refs
paste ref1 ref2 > refs.tsv
sacrebleu refs.tsv --num-refs 2 -i system -m bleu
```

--------------------------------

### Compatibility API Functions

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Use simplified wrapper functions for quick evaluations.

```APIDOC
## Compatibility API Functions

### Description
Use simplified wrapper functions for quick evaluations.

### Method
`corpus_bleu(sys_stream, references, **kwargs)`
`sentence_bleu(hypothesis, references, **kwargs)`
`raw_corpus_bleu(sys_stream, references, **kwargs)`

### Endpoint
N/A (Module functions)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **sys_stream** (iterable) - A list of system outputs (sentences).
- **references** (list of lists of str) - A list of reference translations. Each inner list corresponds to one reference set.
- **hypothesis** (str) - The hypothesis sentence.

### Request Example
```python
import sacrebleu

refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

# Corpus-level BLEU
bleu_score = sacrebleu.corpus_bleu(sys, refs)
print(bleu_score)
# Output: BLEU = 48.53 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18)

# Corpus-level BLEU with custom settings
bleu_lc = sacrebleu.corpus_bleu(
    sys, refs,
    smooth_method='exp',
    lowercase=True,
    tokenize='13a'
)

# Raw corpus BLEU (no tokenization, floor smoothing)
raw_bleu = sacrebleu.raw_corpus_bleu(sys, refs)

# Sentence-level BLEU
sent_bleu = sacrebleu.sentence_bleu(
    "The cat sat on the mat.",
    ["The cat is sitting on the mat.", "A cat sat on the mat."]
)
print(f"Sentence BLEU: {sent_bleu.score:.2f}")
```

### Response
#### Success Response (200)
- **score** (float) - The BLEU score.
- **counts** (list) - Counts of n-gram matches.
- **totals** (list) - Total n-grams in references.
- **precisions** (list) - Precision for each n-gram order.
- **bp** (float) - Brevity penalty.
- **sys_len** (int) - Length of the system output.
- **ref_len** (int) - Length of the reference translation.

#### Response Example
```json
{
  "score": 48.53,
  "counts": [17, 13, 10, 7],
  "totals": [17, 16, 15, 14],
  "precisions": [100.0, 81.25, 66.67, 50.0],
  "bp": 0.943,
  "sys_len": 17,
  "ref_len": 18
}
```
```

--------------------------------

### Evaluate Multiple Systems with SacreBLEU (Table Format)

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Evaluates multiple system output files against a test set and language pair, displaying results in a formatted table. Supports various metrics like BLEU and chrF. The input can be a shell glob matching multiple files.

```bash
$ sacrebleu -t wmt17 -l en-de -i newstest2017.online-* -m bleu chrf
```

--------------------------------

### Scoring with Custom Reference Files in SacreBLEU

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Instructions on how to use SacreBLEU with your own reference files when standard test sets are not applicable. It details saving references to a text file and scoring system outputs against them using either positional arguments or STDIN.

```bash
# Let's save the reference to a text file
$ sacrebleu -t wmt17 -l en-de --echo ref > ref.detok.txt
```

```bash
# Option 1: Pass the reference file as a positional argument to sacreBLEU
$ sacrebleu ref.detok.txt -i output.detok.txt -m bleu -b -w 4
20.7965
```

```bash
# Option 2: Redirect the system into STDIN (Compatible with multi-bleu.perl way of doing things)
$ cat output.detok.txt | sacrebleu ref.detok.txt -m bleu -b -w 4
20.7965
```

--------------------------------

### Compute BLEU with Language-Specific Tokenizers

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Demonstrates how to compute BLEU scores using specific tokenizers like ja-mecab, either explicitly via flags or implicitly by providing a language pair.

```bash
# Using default 13a tokenizer
sacrebleu kyoto-test.ref.ja -i kyoto-test.hyp.ja -b

# Using ja-mecab explicitly
sacrebleu kyoto-test.ref.ja -i kyoto-test.hyp.ja --tokenize ja-mecab -b

# Using language-pair for automatic tokenizer selection
sacrebleu kyoto-test.ref.ja -i kyoto-test.hyp.ja -l en-ja -b
```

--------------------------------

### Manage and Access SacreBLEU Datasets Programmatically

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Provides functions to list, query, and download built-in test sets within the SacreBLEU library. It demonstrates how to retrieve language pairs, file paths, and dataset metadata.

```python
import sacrebleu

# List all available test sets
test_sets = sacrebleu.get_available_testsets()
print(test_sets[:5])
# Output: ['mtedx/test', 'wmt08', 'wmt09', 'wmt10', 'wmt11']

# Get language pairs for a test set
lang_pairs = sacrebleu.get_langpairs_for_testset('wmt17')
print(lang_pairs)
# Output: ['cs-en', 'de-en', 'en-cs', 'en-de', 'en-fi', ...]

# Download and get source file path
source_file = sacrebleu.get_source_file('wmt17', 'en-de')
print(source_file)
# Output: /home/user/.sacrebleu/wmt17/en-de.src

# Get reference file paths
ref_files = sacrebleu.get_reference_files('wmt17', 'en-de')
print(ref_files)
# Output: ['/home/user/.sacrebleu/wmt17/en-de.ref']

# Download a test set explicitly
sacrebleu.download_test_set('wmt17', 'en-de')

# Read files with smart_open (handles .gz compression)
with sacrebleu.smart_open(source_file) as f:
    lines = f.readlines()
    print(f"Number of segments: {len(lines)}")

# Access dataset metadata
from sacrebleu.dataset import DATASETS
dataset = DATASETS['wmt17']
print(dataset.description)
```

--------------------------------

### Using Multiple Metrics

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Demonstrates how to compute BLEU, chrF, and TER scores simultaneously with default and custom settings.

```APIDOC
## Compute Multiple Metrics

### Description
Compute BLEU, chrF, and TER scores with default settings.

### Method
CLI Command

### Endpoint
N/A

### Parameters
#### Command Line Arguments
- **-t, --test-set** (string) - Required - The test set to use (e.g., wmt17).
- **-l, --lang-pair** (string) - Required - The language pair (e.g., en-de).
- **-i, --input** (string) - Required - Path to the input file containing detokenized hypothesis.
- **-m, --metrics** (string) - Required - Comma-separated list of metrics to compute (e.g., bleu,chrf,ter).

### Request Example
```bash
sacrebleu -t wmt17 -l en-de -i output.detok.txt -m bleu chrf ter
```

### Response
#### Success Response (Output)
- **Metric Scores** (string) - The computed scores for each requested metric, including their configurations.

#### Response Example
```
        BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 20.8 <stripped>
      chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.0.0 = 52.0
TER|nrefs:1|case:lc|tok:tercom|norm:no|punct:yes|asian:no|version:2.0.0 = 69.0
```
```

```APIDOC
## Compute Metrics with Custom chrF++

### Description
Compute BLEU, chrF, and TER scores, enabling the chrF++ metric by specifying word order.

### Method
CLI Command

### Endpoint
N/A

### Parameters
#### Command Line Arguments
- **-t, --test-set** (string) - Required - The test set to use (e.g., wmt17).
- **-l, --lang-pair** (string) - Required - The language pair (e.g., en-de).
- **-i, --input** (string) - Required - Path to the input file containing detokenized hypothesis.
- **-m, --metrics** (string) - Required - Comma-separated list of metrics to compute (e.g., bleu,chrf,ter).
- **--chrf-word-order** (integer) - Optional - Sets the word n-gram order for chrF. Setting to 2 enables chrF++.

### Request Example
```bash
sacrebleu -t wmt17 -l en-de -i output.detok.txt -m bleu chrf ter --chrf-word-order 2
```

### Response
#### Success Response (Output)
- **Metric Scores** (string) - The computed scores for each requested metric, including their configurations.

#### Response Example
```
        BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 20.8 <stripped>
    chrF2++|nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.0.0 = 49.0
TER|nrefs:1|case:lc|tok:tercom|norm:no|punct:yes|asian:no|version:2.0.0 = 69.0
```
```

--------------------------------

### Check Available Metadata Fields in SacreBLEU

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

This command is used to discover the available metadata fields for a specific test set and language pair. It shows the possible values for the `--echo` argument.

```bash
$ sacrebleu -t wmt21 -l en-de --echo asdf
```

--------------------------------

### Evaluate Multiple Systems with SacreBLEU (LaTeX Format)

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Evaluates multiple system output files and formats the results in LaTeX table syntax. This is useful for generating reports or publications. It uses the same command-line arguments as the default table format but specifies the output format with '-f latex'.

```bash
$ sacrebleu -t wmt17 -l en-de -i newstest2017.online-* -m bleu chrf -f latex
```

--------------------------------

### Output Metadata with SacreBLEU

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

This command demonstrates how to output additional metadata associated with a test set, such as source sentences, document IDs, and references. The output is tab-separated.

```bash
$ sacrebleu -t wmt21 -l en-de --echo src docid ref | head 2
```

--------------------------------

### CLI Multi-system Evaluation

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Evaluate multiple system outputs simultaneously using specified metrics and output formats.

```APIDOC
## CLI: Multi-system Evaluation

### Description
Evaluates multiple system output files against a test set and language pair, returning a formatted table of scores.

### Method
CLI Command

### Endpoint
sacrebleu -t [testset] -l [langpair] -i [files] -m [metrics]

### Parameters
#### Query Parameters
- **-t** (string) - Required - The test set identifier (e.g., wmt17).
- **-l** (string) - Required - The language pair (e.g., en-de).
- **-i** (string) - Required - Shell glob or list of system output files.
- **-m** (string) - Required - Metrics to compute (e.g., bleu, chrf).
- **-f** (string) - Optional - Output format (e.g., text, latex).

### Request Example
$ sacrebleu -t wmt17 -l en-de -i newstest2017.online-* -m bleu chrf

### Response
#### Success Response (200)
- **Table** (text/latex) - A formatted table containing system names and their respective metric scores.
```

--------------------------------

### Version Signatures

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Explanation of SacreBLEU's version signatures for reproducibility.

```APIDOC
## Understanding Version Signatures

### Description
SacreBLEU generates version strings for reproducibility, detailing the exact configuration used for metric computation.

### Method
Automatic Output

### Endpoint
N/A

### Parameters
N/A

### Request Example
(See Response Example)

### Response
#### Success Response (Output)
- **Version Signature** (string) - A string representing the metric, its parameters, and the SacreBLEU version used (e.g., `BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0`).

#### Response Example
```
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0
```
```

--------------------------------

### Outputting Metadata

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

How to output additional metadata associated with test sets.

```APIDOC
## Outputting Test Set Metadata

### Description
Retrieve metadata fields like source, document ID, or reference for specific test sets.

### Method
CLI Command

### Endpoint
N/A

### Parameters
#### Command Line Arguments
- **-t, --test-set** (string) - Required - The test set to use (e.g., wmt21).
- **-l, --lang-pair** (string) - Required - The language pair (e.g., en-de).
- **--echo** (string) - Required - A comma-separated list of metadata fields to output (e.g., src,docid,ref).
- **-i, --input** (string) - Optional - Path to the input file. If not provided, reads from stdin.

### Request Example
```bash
sacrebleu -t wmt21 -l en-de --echo src,docid,ref | head 2
```

### Response
#### Success Response (Output)
- **Tab-Separated Values** (string) - Metadata fields output as tab-separated columns.

#### Response Example
```
Couple MACED at California dog park for not wearing face masks while having lunch (VIDEO) - RT USA News	rt.com.131279	Paar in Hundepark in Kalifornien mit Pfefferspray besprüht, weil es beim Mittagessen keine Masken trug (VIDEO) - RT USA News
There's mask-shaming and then there's full on assault.	rt.com.131279	Masken-Shaming ist eine Sache, Körperverletzung eine andere.
```
```

```APIDOC
## Listing Available Metadata Fields

### Description
Check which metadata fields are available for a given test set and language pair.

### Method
CLI Command

### Endpoint
N/A

### Parameters
#### Command Line Arguments
- **-t, --test-set** (string) - Required - The test set to use (e.g., wmt21).
- **-l, --lang-pair** (string) - Required - The language pair (e.g., en-de).
- **--echo** (string) - Required - Any string value (e.g., 'asdf') to trigger the listing of available fields.

### Request Example
```bash
sacrebleu -t wmt21 -l en-de --echo asdf
```

### Response
#### Success Response (Output)
- **Available Fields** (string) - A message listing the available metadata fields.

#### Response Example
```
sacreBLEU: No such field asdf in test set wmt21 for language pair en-de.
sacreBLEU: available fields for wmt21/en-de: src, ref:A, ref, docid, origlang
```
```

--------------------------------

### SacreBLEU Compatibility API Functions

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Provides simplified wrapper functions for quick evaluations using SacreBLEU. This includes corpus-level BLEU and TER, with options for custom settings like smoothing methods, case sensitivity, and tokenization. Sentence-level BLEU is also demonstrated.

```python
import sacrebleu

refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

# Corpus-level BLEU
bleu_score = sacrebleu.corpus_bleu(sys, refs)
print(bleu_score)
# Output: BLEU = 48.53 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18)

# Corpus-level BLEU with custom settings
bleu_lc = sacrebleu.corpus_bleu(
    sys, refs,
    smooth_method='exp',
    lowercase=True,
    tokenize='13a'
)

# Raw corpus BLEU (no tokenization, floor smoothing)
raw_bleu = sacrebleu.raw_corpus_bleu(sys, refs)

# Sentence-level BLEU
sent_bleu = sacrebleu.sentence_bleu(
    "The cat sat on the mat.",
    ["The cat is sitting on the mat.", "A cat sat on the mat."]
)
print(f"Sentence BLEU: {sent_bleu.score:.2f}")
```

--------------------------------

### Perform Paired Bootstrap Resampling with SacreBLEU

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

This command executes a paired bootstrap resampling test to compare a baseline system against multiple candidate systems using BLEU and chrF metrics. It identifies statistically significant differences between the systems based on a 0.05 significance threshold.

```bash
sacrebleu -t wmt17 -l en-de -i newstest2017.LIUM-NMT.4900.en-de newstest2017.online-* -m bleu chrf --paired-bs
```

--------------------------------

### Perform Multi-System Evaluation

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Evaluates multiple translation systems simultaneously and generates formatted output tables, including support for LaTeX.

```bash
sacrebleu -t wmt17 -l en-de -i system_a.txt system_b.txt system_c.txt -m bleu chrf
sacrebleu -t wmt17 -l en-de -i system*.txt -m bleu chrf -f latex
```

--------------------------------

### CLI Confidence Intervals

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Calculate confidence intervals for system scores using bootstrap resampling.

```APIDOC
## CLI: Confidence Intervals

### Description
Enables bootstrap resampling to calculate the true mean and 95% confidence interval for a system's score.

### Method
CLI Command

### Endpoint
sacrebleu -t [testset] -l [langpair] -i [file] -m [metrics] --confidence

### Parameters
#### Query Parameters
- **--confidence** (flag) - Required - Enables confidence interval calculation.
- **--confidence-n** (integer) - Optional - Number of bootstrap resamples (default: 1000).

### Request Example
$ sacrebleu -t wmt17 -l en-de -i output.txt -m bleu --confidence

### Response
#### Success Response (200)
- **Output** (string) - Displays the system score, estimated mean, and 95% confidence interval.
```

--------------------------------

### Metric-Specific Arguments

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Details on how to access and use metric-specific arguments for customization.

```APIDOC
## Accessing Metric-Specific Arguments

### Description
Metric-specific arguments can be viewed using the `--help` flag for each metric.

### Method
CLI Command

### Endpoint
N/A

### Parameters
#### Command Line Arguments
- **--help** (string) - Optional - Specify a metric name to see its specific arguments (e.g., `--help bleu`).

### Request Example
```bash
sacrebleu --help bleu
sacrebleu --help chrf
sacrebleu --help ter
```

### Response
#### Success Response (Output)
- **Metric Help Text** (string) - Detailed explanation of available arguments for the specified metric.

#### Response Example (Partial for BLEU)
```
BLEU related arguments:
  --smooth-method {none,floor,add-k,exp}, -s {none,floor,add-k,exp}
                        Smoothing method: exponential decay, floor (increment zero counts), add-k (increment num/denom by k for n>1), or none. (Default: exp)
  --smooth-value BLEU_SMOOTH_VALUE, -sv BLEU_SMOOTH_VALUE
                        The smoothing value. Only valid for floor and add-k. (Defaults: floor: 0.1, add-k: 1)
  --tokenize {none,zh,13a,char,intl,ja-mecab,ko-mecab}, -tok {none,zh,13a,char,intl,ja-mecab,ko-mecab}
                        Tokenization method to use for BLEU. If not provided, defaults to `zh` for Chinese, `ja-mecab` for Japanese, `ko-mecab` for Korean and `13a` (mteval) otherwise.
  --lowercase, -lc      If True, enables case-insensitivity. (Default: False)
  --force               Insist that your tokenized input is actually detokenized.
```
```

--------------------------------

### chrF / chrF++ Metric Configuration

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Options for configuring the chrF and chrF++ metrics, including whitespace handling, case sensitivity, and word order.

```APIDOC
## chrF / chrF++ Metric Configuration

### Description
Configuration options for the chrF and chrF++ metrics, focusing on character n-gram computation.

### Method
N/A (Command-line tool)

### Endpoint
N/A

### Parameters
#### Command-line Flags
- `--chrf-whitespace` (flag) - Preserves whitespace characters when computing character n-grams.
- `--chrf-lowercase` (flag) - Computes case-insensitive chrF scores.
- `--chrf-word-order` (integer) - Enables character n-grams with word order. Pass `2` for chrF++.

### Request Example
```bash
sacrebleu --chrf-lowercase --chrf-word-order 2 my_references.txt -i my_hypotheses.txt
```

### Response
#### Success Response (Output)
- chrF/chrF++ score (float) - The computed score.

#### Response Example
```
75.0
```
```

--------------------------------

### Using the SacreBLEU Compatibility API

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Shows the usage of the legacy wrapper functions for computing corpus-level BLEU scores. Note that this API may be deprecated in future versions.

```python
import sacrebleu

refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

score = sacrebleu.corpus_bleu(sys, refs)
print(score)
```

--------------------------------

### Handle Variable Number of References for BLEU Score Calculation

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Shows how to calculate BLEU scores when different sentences have varying numbers of references. Placeholders like '' or None can be used for missing references.

```python
from sacrebleu.metrics import BLEU

# Some sentences have fewer references (use '' or None as placeholder)
refs = [
    # First reference set (first sentence has no ref here)
    ['', 'It was not unexpected.', 'The man bit him first.'],
    # Second reference set
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

bleu = BLEU()
result = bleu.corpus_score(sys, refs)
print(result)
# Output: BLEU = 29.44 82.4/42.9/27.3/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)

print(bleu.get_signature())
# Output: nrefs:var|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0
# Note: nrefs:var indicates variable references
```

--------------------------------

### Extract N-grams for Custom Metric Implementations

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Illustrates how to use helper functions from sacrebleu.metrics.helpers to extract word and character n-grams from text. These are useful for implementing custom evaluation metrics.

```python
from sacrebleu.metrics.helpers import extract_word_ngrams, extract_char_ngrams

# Extract word n-grams
text = "The quick brown fox jumps over the lazy dog"
words = text.split()

unigrams = extract_word_ngrams(words, 1)
print(dict(unigrams))
# Output: {('The',): 1, ('quick',): 1, ('brown',): 1, ...}

bigrams = extract_word_ngrams(words, 2)
print(dict(bigrams))
# Output: {('The', 'quick'): 1, ('quick', 'brown'): 1, ...}

# Using the public API
from sacrebleu import extract_word_ngrams, extract_char_ngrams

word_bigrams = extract_word_ngrams(words, 2)
char_trigrams = extract_char_ngrams(text, 3)
```

--------------------------------

### Calculate Corpus-level and Sentence-level chrF and TER Scores with SacreBLEU

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Demonstrates how to compute chrF and TER scores at both corpus and sentence levels using the sacrebleu library. It shows basic usage and how to apply options for TER calculation.

```python
import sacrebleu

# Corpus-level chrF
chrf_score = sacrebleu.corpus_chrf(sys, refs)
print(chrf_score)
# Output: chrF2 = 59.73

# chrF++ (word_order=2)
chrfpp_score = sacrebleu.corpus_chrf(sys, refs, word_order=2)

# Sentence-level chrF
sent_chrf = sacrebleu.sentence_chrf(
    "The cat sat on the mat.",
    ["The cat is sitting on the mat."]
)

# Corpus-level TER
ter_score = sacrebleu.corpus_ter(sys, refs)
print(ter_score)
# Output: TER = 31.2

# TER with options
ter_normalized = sacrebleu.corpus_ter(
    sys, refs,
    normalized=True,
    case_sensitive=True
)

# Sentence-level TER
sent_ter = sacrebleu.sentence_ter(
    "The cat sat on the mat.",
    ["The cat is sitting on the mat."]
)
```

--------------------------------

### Use Python API for Corpus Scoring

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Demonstrates how to use the BLEU class in Python to compute scores programmatically with custom configurations.

```python
from sacrebleu.metrics import BLEU
refs = [['The dog bit the man.', 'It was not unexpected.'], ['The dog had bit the man.', 'No one was surprised.']]
sys = ['The dog bit the man.', "It wasn't surprising."]
bleu = BLEU()
result = bleu.corpus_score(sys, refs)
print(result.score)

bleu_custom = BLEU(lowercase=True, tokenize='intl', smooth_method='floor')
result = bleu_custom.corpus_score(sys, refs)
```

--------------------------------

### Handling Variable Number of References in BLEU

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Demonstrates how to handle datasets where some hypotheses lack corresponding references by using empty strings. This changes the metric signature to nrefs:var.

```python
from sacrebleu.metrics import BLEU, CHRF, TER

refs = [
    ['', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

bleu = BLEU()
score = bleu.corpus_score(sys, refs)
print(bleu.get_signature())
```

--------------------------------

### BLEU Metric Configuration

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

Details on configuring the BLEU metric in SacreBLEU, including case sensitivity and tokenizer options.

```APIDOC
## BLEU Metric Configuration

### Description
Configuration options for the BLEU metric in SacreBLEU, including case sensitivity and tokenizer selection.

### Method
N/A (Command-line tool)

### Endpoint
N/A

### Parameters
#### Command-line Flags
- `--lowercase` (flag) - Enables case-insensitive BLEU computation.
- `--tokenize` (string) - Specifies the tokenizer to use. Defaults to `13a`.
  - Available tokenizers: `none`, `char`, `intl`, `zh`, `ja-mecab`, `ko-mecab`, `flores101`, `flores200`, `spBLEU-1K`.
- `-l` or `--language-pair` (string) - Language pair (e.g., `en-ja`). Automatically selects appropriate tokenizers for `zh`, `ja`, `ko`.

### Request Example
```bash
sacrebleu kyoto-test.ref.ja -i kyoto-test.hyp.ja --tokenize ja-mecab -b
```

### Response
#### Success Response (Output)
- BLEU score (float) - The computed BLEU score.

#### Response Example
```
14.5
```
```

--------------------------------

### Apply Language-Specific Tokenization

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Configures tokenizers for various languages including Japanese, Chinese, and Korean. Supports character-level and specialized tokenization like SentencePiece.

```bash
sacrebleu kyoto-test.ref.ja -i kyoto-test.hyp.ja --tokenize ja-mecab -b
sacrebleu ref.zh -i hyp.zh --tokenize zh -b
sacrebleu ref.ko -i hyp.ko --tokenize ko-mecab -b
sacrebleu ref.txt -i hyp.txt --tokenize char -b
sacrebleu ref.txt -i hyp.txt --tokenize flores200 -b
```

--------------------------------

### Configure BLEU Smoothing Methods

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Applies different smoothing techniques to BLEU scores to handle zero n-gram counts. This is essential for short sentences or small test sets.

```bash
sacrebleu -t wmt17 -l en-de -i output.detok.txt -m bleu --smooth-method floor --smooth-value 0.1 -b
sacrebleu -t wmt17 -l en-de -i output.detok.txt -m bleu --smooth-method add-k --smooth-value 1 -b
```

--------------------------------

### Bootstrap Resampling for Confidence Intervals

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Perform bootstrap resampling to calculate confidence intervals for metrics.

```APIDOC
## Bootstrap Resampling for Confidence Intervals

### Description
Performs bootstrap resampling to calculate confidence intervals for SacreBLEU scores, providing a measure of statistical significance.

### Method
`metric.corpus_score(sys, references, n_bootstrap=1000)`

### Parameters
- **sys** (list of str) - System outputs.
- **references** (list of list of str, optional) - References. If `None`, uses references provided during metric initialization.
- **n_bootstrap** (int, optional) - The number of bootstrap samples to generate. Defaults to 1000.

### Configuration
- **SACREBLEU_SEED** (str, optional) - Set an environment variable to control the random seed for reproducibility.

### Request Example
```python
from sacrebleu.metrics import BLEU
import os

# Set random seed for reproducibility (optional)
os.environ['SACREBLEU_SEED'] = '42'

refs = [['Reference sentence one.', 'Reference sentence two.']]
sys = ['Hypothesis sentence one.', 'Hypothesis sentence two.']

bleu = BLEU(references=refs)

# Compute corpus score with bootstrap confidence intervals
score = bleu.corpus_score(sys, references=None, n_bootstrap=1000)

# Access bootstrap statistics
print(f"Score: {score.score:.2f}")
print(f"Mean: {score.mean:.2f}" if hasattr(score, 'mean') else "No bootstrap")
print(f"CI: ±{score.ci:.2f}" if hasattr(score, 'ci') else "No CI")
```

### Response Attributes
- **score** (float) - The primary score.
- **mean** (float, optional) - The mean score from bootstrap samples.
- **ci** (float, optional) - The confidence interval (half-width) from bootstrap samples.
```

--------------------------------

### Compute BLEU, chrF, and TER Metrics with SacreBLEU

Source: https://github.com/mjpost/sacrebleu/blob/master/README.md

This command computes the BLEU, chrF, and TER scores for a given output file against a reference test set. It uses default settings for each metric.

```bash
$ sacrebleu -t wmt17 -l en-de -i output.detok.txt -m bleu chrf ter
```

--------------------------------

### Perform Bootstrap Resampling for Confidence Intervals

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Demonstrates how to compute BLEU scores with bootstrap resampling to obtain confidence intervals, providing a measure of statistical significance for the evaluation results.

```python
from sacrebleu.metrics import BLEU
import os

# Set random seed for reproducibility (optional)
os.environ['SACREBLEU_SEED'] = '42'

refs = [['Reference sentence one.', 'Reference sentence two.']]
sys = ['Hypothesis sentence one.', 'Hypothesis sentence two.']

bleu = BLEU(references=refs)

# Compute corpus score with bootstrap confidence intervals
score = bleu.corpus_score(sys, references=None, n_bootstrap=1000)

# Access bootstrap statistics
print(f"Score: {score.score:.2f}")
print(f"Mean: {score.mean:.2f}" if hasattr(score, 'mean') else "No bootstrap")
print(f"CI: ±{score.ci:.2f}" if hasattr(score, 'ci') else "No CI")
```

--------------------------------

### SacreBLEU BLEU Corpus Score with Caching

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Demonstrates pre-caching references for efficient corpus-level BLEU scoring across multiple system outputs. This approach speeds up evaluations when the same references are used repeatedly.

```python
from sacrebleu.metrics import BLEU

refs = [['This is the first reference.', 'This is the second reference.']]
sys = ['This is the system output.']
another_sys = ['Another system output.']

# Pre-cache references for multiple system evaluations
bleu_cached = BLEU(references=refs)
result1 = bleu_cached.corpus_score(sys, references=None)
result2 = bleu_cached.corpus_score(another_sys, references=None)  # Faster
```

--------------------------------

### Dataset Management Functions

Source: https://context7.com/mjpost/sacrebleu/llms.txt

Access and manage built-in test sets programmatically.

```APIDOC
## Dataset Management Functions

### Description
Access and manage built-in test sets programmatically, including listing available sets, getting language pairs, and downloading data.

### Functions

#### `sacrebleu.get_available_testsets()`

- **Description**: Lists all available test sets.
- **Returns**: List of strings representing test set names.

#### `sacrebleu.get_langpairs_for_testset(testset)`

- **Description**: Gets language pairs available for a given test set.
- **Parameters**: 
  - **testset** (str) - The name of the test set.
- **Returns**: List of strings representing language pairs (e.g., 'en-de').

#### `sacrebleu.get_source_file(testset, langpair)`

- **Description**: Gets the path to the source file for a specific test set and language pair.
- **Parameters**: 
  - **testset** (str) - The name of the test set.
  - **langpair** (str) - The language pair (e.g., 'en-de').
- **Returns**: String path to the source file.

#### `sacrebleu.get_reference_files(testset, langpair)`

- **Description**: Gets the paths to the reference files for a specific test set and language pair.
- **Parameters**: 
  - **testset** (str) - The name of the test set.
  - **langpair** (str) - The language pair (e.g., 'en-de').
- **Returns**: List of strings representing paths to reference files.

#### `sacrebleu.download_test_set(testset, langpair)`

- **Description**: Downloads a specific test set and language pair.
- **Parameters**: 
  - **testset** (str) - The name of the test set.
  - **langpair** (str) - The language pair (e.g., 'en-de').

#### `sacrebleu.smart_open(filepath)`

- **Description**: Opens a file using `smart_open`, handling potential `.gz` compression.
- **Parameters**: 
  - **filepath** (str) - The path to the file.
- **Returns**: A file-like object.

#### Accessing Dataset Metadata

- **Description**: Access metadata for a specific dataset.
- **Code**: 
```python
from sacrebleu.dataset import DATASETS
dataset = DATASETS['wmt17']
print(dataset.description)
```

### Examples

#### List available test sets
```python
import sacrebleu
test_sets = sacrebleu.get_available_testsets()
print(test_sets[:5])
```
*Output: `['mtedx/test', 'wmt08', 'wmt09', 'wmt10', 'wmt11']`*

#### Get language pairs for a test set
```python
lang_pairs = sacrebleu.get_langpairs_for_testset('wmt17')
print(lang_pairs)
```
*Output: `['cs-en', 'de-en', 'en-cs', 'en-de', 'en-fi', ...]`*

#### Get source file path
```python
source_file = sacrebleu.get_source_file('wmt17', 'en-de')
print(source_file)
```
*Output: `/home/user/.sacrebleu/wmt17/en-de.src`*

#### Get reference file paths
```python
ref_files = sacrebleu.get_reference_files('wmt17', 'en-de')
print(ref_files)
```
*Output: `['/home/user/.sacrebleu/wmt17/en-de.ref']`*

#### Download a test set
```python
sacrebleu.download_test_set('wmt17', 'en-de')
```

#### Read files with smart_open
```python
with sacrebleu.smart_open(source_file) as f:
    lines = f.readlines()
    print(f"Number of segments: {len(lines)}")
```
```