OpusFilter

https://helsinki-nlp.github.io/OpusFilter
Admin
OpusFilter is a tool for filtering and combining parallel corpora using corpus preprocessing...
Tokens:19,535
Snippets:168
Trust Score:9.7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Filtering with Custom Parameter Options

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

Example of performing the same filtering task using individual command-line options instead of a single JSON object.

```bash
opusfilter-cmd filter --inputs wmt.fi.gz wmt.en.gz --outputs wmt_filtered.fi.gz wmt_filtered.en.gz --filters '[{"LengthFilter": {"unit": "word", "min_length": 1, "max_length": 100}}]'
```

--------------------------------

### Reusable Filter Configuration with YAML Anchors

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

This example demonstrates using YAML anchors (`&`) and references (`*`) to define and reuse a set of filters across different data sources. It applies the same length and length ratio filters to both ParaCrawl and WMT-News data separately.

```yaml
  - type: filter
    parameters:
      inputs:
      - paracrawl.fi.gz
      - paracrawl.en.gz
      outputs:
      - paracrawl_filtered.fi.gz
      - paracrawl_filtered.en.gz
      filters: &myfilters
        - LengthFilter:
            unit: word
            min_length: 1
            max_length: 100

        - LengthRatioFilter:
            unit: word
            threshold: 3

  - type: filter
    parameters:
      inputs:
      - wmt.fi.gz
      - wmt.en.gz
      outputs:
      - wmt_filtered.fi.gz
      - wmt_filtered.en.gz
      filters: *myfilters

```

--------------------------------

### Implement a custom UppercaseFilter

Source: https://helsinki-nlp.github.io/OpusFilter/filters/custom_filters.html

Example of a custom filter that calculates the proportion of uppercase letters in sentences. Requires inheriting from FilterABC and implementing the score and accept methods.

```python
import opusfilter

class UppercaseFilter(opusfilter.FilterABC):

	score_direction = opusfilter.CLEAN_LOW
	accept_threshold = 1 + 10**-6
	reject_threshold = 0

    def __init__(self, threshold=0.5, **kwargs):
        self.threshold = threshold
        super().__init__(**kwargs)

    def uppercase_ratio(self, sentence):
        length = len(sentence)
        if length > 0:
            return sum(1 for char in sent if char.isupper()) / length
        return 0

    def score(self, pairs):
        for pair in pairs:
            yield [self.uppercase_ratio(sentence) for sentence in pair]

    def accept(self, score):
        return all(ratio < self.threshold for ratio in score)
```

--------------------------------

### Filtering with JSON Parameters

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

Example of performing a word length filter using a single JSON object passed via the --parameters flag.

```bash
opusfilter-cmd filter --parameters {"inputs": ["wmt.fi.gz", "wmt.en.gz"], "outputs": ["wmt_filtered.fi.gz", "wmt_filtered.en.gz"], "filters": [{"LengthFilter": {"unit": "word", "min_length": 1, "max_length": 100}}]}
```

--------------------------------

### head

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Take the first n lines from files.

```APIDOC
## head

### Description
Take the first n lines from files.

### Parameters
- **inputs** (list) - Required - A list of input files
- **outputs** (list) - Required - A list of output files
- **n** (integer) - Required - Number of output lines
```

--------------------------------

### Visualize configuration with opusfilter-diagram

Source: https://helsinki-nlp.github.io/OpusFilter/command_line_tools.html

Generates a directed acyclic graph from a configuration file using graphviz.

```bash
opusfilter-diagram [--rankdir {TB,LR}] FILE FILE
```

--------------------------------

### preprocess

Source: https://helsinki-nlp.github.io/OpusFilter/functions/preprocessing_text.html

Applies a series of preprocessors to input files and writes the results to output files.

```APIDOC
## preprocess

### Description
Preprocess text with a combination of preprocessors.

### Parameters
- **inputs** (list) - Required - Input files for segments to preprocess.
- **outputs** (list) - Required - Output files for preprocessed segments.
- **n_jobs** (int) - Optional - Number of sub processes to parallel run jobs. If not set, the default value is `default_n_jobs` in `common` section.
- **preprocessors** (list) - Required - A list of dictionaries, each representing one preprocessor. The top level should include a key defining the class name (e.g., `WhitespaceNormalizer`) and optionally a `module` key for custom preprocessors.
```

--------------------------------

### Run OpusFilter command

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

Execute the main opusfilter script using a YAML configuration file.

```bash
opusfilter [--overwrite] [--last LAST] [--single SINGLE] [--n-jobs N_JOBS] CONFIG
```

--------------------------------

### split

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Split files into two parts using hash-based deterministic splitting.

```APIDOC
## split

### Description
Split files to two parts giving the approximative proportions as fractions.

### Parameters
- **inputs** (list) - Required - Input file(s)
- **outputs** (list) - Required - Output file(s) for selected lines
- **outputs_2** (list) - Optional - Output file(s) for the rest of the lines
- **divisor** (integer) - Required - Divisor for the modulo operation
- **threshold** (integer) - Optional - Threshold for the output of the modulo operation (default 1)
- **compare** (list/string) - Optional - Select files to use for hash operation (default 'all')
- **hash** (string) - Optional - Select hash algorithm (default 'xxh64')
- **seed** (integer) - Optional - Integer seed for the hash algorithm (default 0)
```

--------------------------------

### download

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Download a file from a URL.

```APIDOC
## download

### Description
Download a file from URL.

### Parameters
- **url** (string) - Required - URL for the file to download
- **output** (string) - Required - Output file
```

--------------------------------

### slice

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Take a slice of lines from files.

```APIDOC
## slice

### Description
Take slice of lines from files.

### Parameters
- **inputs** (list) - Required - A list of input files
- **outputs** (list) - Required - A list of output files
- **start** (integer) - Optional - Start index (default 0)
- **stop** (integer) - Optional - Stop index (default null)
- **step** (integer) - Optional - Step size (default 1)
```

--------------------------------

### BPESegmentation Configuration

Source: https://helsinki-nlp.github.io/OpusFilter/preprocessors/bpe_segmentation.html

Details regarding the parameters available for configuring the BPE segmentation process.

```APIDOC
## BPESegmentation

### Description
Split words into subword units using a BPE model based on the Sennrich et al. (2016) approach.

### Parameters
- **model** (file) - Required - A model file containing BPE codes.
- **merges** (int) - Optional - Number of BPE operations to use (default: -1, use all).
- **separator** (string) - Optional - Separator between non-final subword units (default: "@@").
- **vocab** (file) - Optional - Vocabulary file; if provided, reverts merge operations that produce an OOV (default: null).
- **glossaries** (list/regex) - Optional - Words matching these will not be affected (default: null).
- **dropout** (float) - Optional - Probability to dropout BPE merge operations (default: 0).
```

--------------------------------

### write

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Writes a specified string into a file.

```APIDOC
## write

### Description
Write a specified string into a file. Useful mostly for testing.

### Parameters
- **output** (string) - Required - Output file
- **data** (any) - Required - Input data to write to the output
```

--------------------------------

### OpusFilter Command Syntax

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

General syntax for executing the opusfilter-cmd script.

```bash
opusfilter-cmd [--overwrite] [--outputdir OUTDIR] FUNCTION [--parameters PARAMETERS] [PARAMETER-OPTION] ...
```

--------------------------------

### train_ngram

Source: https://helsinki-nlp.github.io/OpusFilter/functions/training_language_and_alignment_models.html

Train a character-based varigram language model using VariKN.

```APIDOC
## train_ngram

### Description
Train a character-based varigram language model with VariKN. This model can be used for CrossEntropyFilter and CrossEntropyDifferenceFilter.

### Parameters
- **data** (string) - Required - Input file name for training data
- **model** (string) - Required - Output file name for the model
- **parameters** (object) - Optional - Training options for VariKN and tokenization:
  - **optdata** (string) - Optional - Filename for optimization data
  - **norder** (integer) - Optional - Limit model order
  - **dscale** (float) - Optional - Model size scale factor
  - **dscale2** (float) - Optional - Model size scaling during pruning
  - **arpa** (boolean) - Optional - Output ARPA instead of binary LM
  - **use_3nzer** (boolean) - Optional - Use 3 discounts per order
  - **absolute** (boolean) - Optional - Use absolute discounting
  - **cutoffs** (string) - Optional - Specified cutoffs
  - **segmentation** (object) - Optional - Subword segmentation options
  - **mb** (string) - Optional - Word-internal boundary marking
  - **wb** (string) - Optional - Word boundary tag
```

--------------------------------

### Download Finnish-English ParaCrawl Corpus

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

This configuration downloads a specified release of the Finnish-English ParaCrawl corpus from OPUS and saves its segments to gzipped files. Ensure the output filenames end with .gz for compression.

```yaml
steps:
  - type: opus_read
    parameters:
      corpus_name: ParaCrawl
      source_language: fi
      target_language: en
      release: v4
      preprocessing: raw
      src_output: paracrawl.fi.gz
      tgt_output: paracrawl.en.gz

```

--------------------------------

### product

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Creates a Cartesian product of parallel segments with optional sampling.

```APIDOC
## product

### Description
Create a Cartesian product of parallel segments and optionally sample from them. Useful for combining parallel files of the same language containing alternative translations.

### Parameters
- **inputs** (list) - Required - A list of input files lists
- **outputs** (list) - Required - A list of output files
- **skip_empty** (boolean) - Optional - Skip empty lines (default: true)
- **skip_duplicates** (boolean) - Optional - Skip duplicate lines per language (default: true)
- **k** (integer) - Optional - Sample at most k random items per product (default: null)
- **seed** (integer) - Optional - Seed for the random generator (default: null)
```

--------------------------------

### RegExpSub Tool

Source: https://helsinki-nlp.github.io/OpusFilter/preprocessors/reg_exp_sub.html

This section details the RegExpSub tool for applying multiple regular expression substitutions.

```APIDOC
## RegExpSub

### Description
Run a sequence of arbitrary regular expression substitutions.

### Parameters

#### Patterns
- `patterns` (list) - A list of patterns to use by default.
- `lang_patterns` (dict or list) - A dictionary or list of patterns to use for specific languages.

Multiple substitutions are applied in the given order. The default patterns are replaced with language-specific patterns when the corresponding index (starting from 0) is found in the `lang_patterns` dictionary. The `lang_patterns` argument may also be a list, if you e.g. want to use separate patterns for all languages.

The substitution patterns are 4-tuples containing the regular expression, replacement, count (0 = substitute all) and flags (list of flag constants in the `re` library, e.g. `["I", "A"]`). The regular expressions are first compiled with `re.compile`, and then the substitutions are applied with `re.sub`.
```

--------------------------------

### Join score files with YAML configuration

Source: https://helsinki-nlp.github.io/OpusFilter/functions/using_score_files.html

Use the join type to merge multiple score files into a single output file, optionally nesting values under specific keys.

```yaml
  - type: join
    parameters:
      inputs:
      - scores.jsonl.gz
      - myscores.src
      - myscores.tgt
      keys:
      - null
      - MyScore.src
      - MyScore.tgt
      output: scores-joined.jsonl.gz
```

--------------------------------

### tail

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Take the last n lines from files.

```APIDOC
## tail

### Description
Take the last n lines from files.

### Parameters
- **inputs** (list) - Required - A list of input files
- **outputs** (list) - Required - A list of output files
- **n** (integer) - Required - Number of output lines
```

--------------------------------

### Download and Filter Multiple Corpora with Length Constraints

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

This configuration downloads both ParaCrawl and WMT-News corpora, concatenates them, and then filters the combined data. It applies length and length ratio filters to ensure segments are between 1-100 words and their length ratio is at most 3.

```yaml
steps:
  - type: opus_read
    parameters:
      corpus_name: ParaCrawl
      source_language: fi
      target_language: en
      release: v4
      preprocessing: raw
      src_output: paracrawl.fi.gz
      tgt_output: paracrawl.en.gz

  - type: opus_read
    parameters:
      corpus_name: WMT-News
      source_language: fi
      target_language: en
      release: v2019
      preprocessing: raw
      src_output: wmt.fi.gz
      tgt_output: wmt.en.gz

  - type: concatenate
    parameters:
      inputs:
      - paracrawl.fi.gz
      - wmt.fi.gz
      output: all.fi.gz

  - type: concatenate
    parameters:
      inputs:
      - paracrawl.en.gz
      - wmt.en.gz
      output: all.en.gz

  - type: filter
    parameters:
      inputs:
      - all.fi.gz
      - all.en.gz
      outputs:
      - filtered.fi.gz
      - filtered.en.gz
      filters:
        - LengthFilter:
            unit: word
            min_length: 1
            max_length: 100

        - LengthRatioFilter:
            unit: word
            threshold: 3

```

--------------------------------

### RegExpFilter Configuration

Source: https://helsinki-nlp.github.io/OpusFilter/filters/special_character_and_similarity_filters.html

Configure the RegExpFilter to process segments based on provided regular expressions.

```APIDOC
## RegExpFilter

Filter out segments that match (or do not match) an arbitrary regular expression.

### Parameters

- **regexps** (string | list[string]) - Required - A regular expression or a list of expressions to match.
- **accept_match** (boolean) - Optional - Accept matching segments instead of rejecting (default `false`).

### Usage Details

You can either provide a single regexp or one for each language in the parallel data. If `accept_match` is `false`, the pair is accepted only if none of the segment match the corresponding regular expression. If `accept_match` is `true`, the pair is accepted only if all segments match the corresponding regular expression.

The `regex` module is used for the regular expressions.
```

--------------------------------

### Configure a custom filter in YAML

Source: https://helsinki-nlp.github.io/OpusFilter/filters/custom_filters.html

How to reference a custom filter module within the OpusFilter configuration steps.

```yaml
steps:

  ...

  - type: filter
    parameters:

      ...

      filters:

        - UppercaseFilter:
            threshold: 0.5
          module: customfilter
```

--------------------------------

### Use Global Constant and Local Target in OpusFilter

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

Utilize a global constant 'source' and a local constant 'target' within a step to construct dynamic file paths. The local 'target' overrides any global definition.

```yaml
steps:
  - type: concatenate
    parameters:
      inputs:
      - !varstr "file1.{source}-{target}.gz"
      - !varstr "file2.{source}-{target}.gz"
      output: !varstr "all.{source}-{target}.gz"
    constants:
      target: fi
```

--------------------------------

### concatenate

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Concatenate two or more text files.

```APIDOC
## concatenate

### Description
Concatenate two or more text files.

### Parameters
- **inputs** (list) - Required - A list of input files
- **output** (string) - Required - Output file
```

--------------------------------

### subset

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Take a random subset from parallel corpus files.

```APIDOC
## subset

### Description
Take a random subset from parallel corpus files.

### Parameters
- **inputs** (list) - Required - Input files
- **outputs** (list) - Required - Output files for storing the subset
- **size** (integer) - Required - Number of lines to select for the subset
- **seed** (integer) - Optional - Seed for the random generator
- **shuffle_subset** (boolean) - Optional - Shuffle the order of the selected lines (default false)
```

--------------------------------

### Define Multiple Variable Choices in OpusFilter

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

Define a list of possible values for the 'target' variable within a step. This will generate multiple substeps, one for each value in the list.

```yaml
steps:
  - type: concatenate
    parameters:
      inputs:
      - !varstr "file1.{source}-{target}.gz"
      - !varstr "file2.{source}-{target}.gz"
      output: !varstr "all.{source}-{target}.gz"
    variables:
      target: [fi, sv]
```

--------------------------------

### OpusFilter Autogen Script Usage

Source: https://helsinki-nlp.github.io/OpusFilter/automatic_configuration.html

Use this script to generate OpusFilter configuration files. It supports various options for specifying input files, language codes, script types, filtering methods, and output configurations.

```bash
usage: opusfilter-autogen [-h] --files TEXTFILE [TEXTFILE ...]
                          [--langs LANGCODE [LANGCODE ...]]
                          [--scripts SCRIPT [SCRIPT ...]]
                          [--method {defaults,percentiles,clustering}]
                          [--sample-size SAMPLE_SIZE]
                          [--noisy-percentile NOISY_PERCENTILE]
                          [--work-dir WORK_DIR] [--inter-dir INTER_DIR]
                          [--plot] [--list-defaults] [--add-filter CLASS JSON]
                          [--overwrite] [-o CONFIGFILE]

Generate initial configuration based on parallel text data

options:
  -h, --help            show this help message and exit
  --files TEXTFILE [TEXTFILE ...]
                        parallel text input file(s)
  --langs LANGCODE [LANGCODE ...]
                        Language codes corresponding to the input files. If
                        omitted, LanguageIDFilters will not be used.
  --scripts SCRIPT [SCRIPT ...]
                        Alphabetic scripts (e.g. Latin) corresponding to the
                        input files. If omitted, CharacterScoreFilter will not
                        be used.
  --method {defaults,percentiles,clustering}
                        Method for selecting filter thresholds (default:
                        clustering)
  --sample-size INT     Max number of sentence pairs used for data-based
                        methods (default 100000)
  --noisy-percentile FLOAT
                        Proportion of the data considered to be noisy; only
                        for percentiles method (default 0.001)
  --clusters INT, -k INT
                        Number of clusters for the clustering method; try
                        increasing if too much data is clustered as noisy
                        (default 2)
  --work-dir WORK_DIR   Location of the source and target files for the
                        generated configuration (default work)
  --inter-dir INTER_DIR
                        Save intermediate files in this directory (use a
                        temporary directory if not given)
  --plot                Show a scatter plot of the clustering and histograms
                        of feature data distributions; only for the clustering
                        method
  --list-defaults       List default filters of the method to the output and
                        quit
  --add-filter CLASS JSON
                        Instead of using default filters, add a filter of
                        CLASS with JSON parameters object ("{}" for default
                        parameters). The class name may be followed by a dot
                        and a unique filter identifier in order to allow
                        multiple filters of the same class. Example: --add-
                        filter LanguageIDFilter.cld2 '{"id_method": "cld2"}'
  --overwrite           Overwrite existing intermediate files
  -o CONFIGFILE, --output CONFIGFILE
                        Output configuration file (default -)

```

--------------------------------

### unzip

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Unzips parallel segments joined in a single file into multiple files.

```APIDOC
## unzip

### Description
Unzip parallel segments joined in a single file into multiple files. Can be used to split Moses-style or tab-separated parallel text files.

### Parameters
- **input** (string) - Required - Input file
- **outputs** (list) - Required - A list of output files
- **separator** (string) - Required - A string separator in the input file
```

--------------------------------

### train_aligment

Source: https://helsinki-nlp.github.io/OpusFilter/functions/training_language_and_alignment_models.html

Train word alignment priors for eflomal.

```APIDOC
## train_aligment

### Description
Train word alignment priors for eflomal. Can be used in WordAlignFilter.

### Parameters
- **src_data** (string) - Required - Input file for the source language
- **tgt_data** (string) - Required - Input file for the target language
- **parameters** (object) - Optional - Training options:
  - **src_tokenizer** (string) - Optional - Tokenizer for source language
  - **tgt_tokenizer** (string) - Optional - Tokenizer for target language
  - **model** (integer) - Optional - eflomal model type
- **output** (string) - Required - Output file name for the priors
- **scores** (string) - Optional - File to write alignment scores
```

--------------------------------

### POST /score

Source: https://helsinki-nlp.github.io/OpusFilter/functions/filtering_and_scoring.html

Calculates filtering scores for lines of parallel data and writes them to an output file.

```APIDOC
## POST /score

### Description
Calculate filtering scores for the lines of parallel data. The output is provided in JSON Lines format, where each line contains a JSON object with filter class names and their corresponding scores.

### Method
POST

### Endpoint
/score

### Parameters
#### Request Body
- **inputs** (list) - Required - Input files for segments to score
- **output** (string) - Required - Output file path for the scores
- **n_jobs** (integer) - Optional - Number of sub-processes to run in parallel. Defaults to common.default_n_jobs
- **filters** (list) - Required - A list of filter configurations to apply

### Response
#### Success Response (200)
- **JSON Lines** (object) - Each line contains a JSON object where keys are filter class names and values are the calculated scores.
```

--------------------------------

### Analyze duplicates with opusfilter-duplicates

Source: https://helsinki-nlp.github.io/OpusFilter/command_line_tools.html

Calculates duplicate statistics or corpus overlap using specified hashing and normalization options.

```bash
opusfilter-duplicates [--overlap FILE [FILE ...]] [--hash HASH] [--letters-only]
[--letter-words-only] [--lowercase] [--tokenizers JSON] FILE [FILE ...]
```

--------------------------------

### Test filters with opusfilter-test

Source: https://helsinki-nlp.github.io/OpusFilter/command_line_tools.html

Evaluates how many segments a filter removes from parallel data and optionally exports removed segments.

```bash
opusfilter-test [--yaml FILE] [--add CLASS JSON] [--removed FILE] FILE [FILE ...]
```

--------------------------------

### Define Global Constant in OpusFilter

Source: https://helsinki-nlp.github.io/OpusFilter/usage.html

Define a constant value 'source' in the 'common' section to be used across all steps.

```yaml
common:
  constants:
    source: en
```

--------------------------------

### Classify Sentence Pairs

Source: https://helsinki-nlp.github.io/OpusFilter/functions/training_and_using_classifiers.html

Uses a trained classifier model to assign cleanness scores or labels to sentence pairs.

```APIDOC
## classify

### Description
Use a classifier model trained with `train_classifier` to assign a cleanness score or label to sentence pairs that have been scored with `score`.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

#### Parameters
- **model** (file) - Required - Classifier model trained with `train_classifier`.
- **scores** (file) - Required - Scores of the sentence pairs to be classified in JSON lines format produced with the `score` function.
- **output_probabilities** (file) - Optional - File to write the cleanness scores to, 1 is cleanest and 0 is noisiest.
- **output_labels** (file) - Optional - File to write the cleanness labels to, 1 is a clean and 0 is a noisy pair.

### Request Example
None

### Response
#### Success Response (200)
None

#### Response Example
None

### Notes
The probabilities and labels are written to the output files line by line, corresponding to the scores on each line in `scores`.
```

--------------------------------

### train_nearest_neighbors

Source: https://helsinki-nlp.github.io/OpusFilter/functions/training_language_and_alignment_models.html

Train unsupervised model to search for nearest neighbors of segments.

```APIDOC
## train_nearest_neighbors

### Description
Train unsupervised model to search for nearest neighbors of segments using sentence embeddings. Can be used in SentenceEmbeddingFilter.

### Parameters
- **inputs** (list) - Required - A list of input files
- **languages** (list) - Required - A list of language codes corresponding to the input files
- **n_neighbors** (integer) - Optional - Default number of neighbors to return
- **algorithm** (string) - Optional - Algorithm used to compute neighbors
- **metric** (string) - Optional - Distance or similarity metric
- **output** (string) - Required - Output file name for the model
```

--------------------------------

### opus_read

Source: https://helsinki-nlp.github.io/OpusFilter/functions/downloading_and_selecting_data.html

Read a corpus from the OPUS corpus collection.

```APIDOC
## opus_read

### Description
Read a corpus from the OPUS corpus collection using the OpusTools interface.

### Parameters
- **corpus_name** (string) - Required - Name of the corpus in OPUS
- **source_language** (string) - Required - Language code for the source language
- **target_language** (string) - Required - Language code for the target language
- **release** (string) - Required - Version of the corpus in OPUS
- **preprocessing** (string) - Required - 'moses' or 'raw' for untokenized and 'xml' for tokenized segments
- **src_output** (string) - Required - Output file for source language
- **tgt_output** (string) - Required - Output file for target language
- **suppress_prompts** (boolean) - Optional - 'false' (default) prompts user to confirm before download, 'true' to download without prompting
```

--------------------------------

### Heliport Filters

Source: https://helsinki-nlp.github.io/OpusFilter/filters/script_and_language_identification_filters.html

Filters based on the HeLI-OTS method.

```APIDOC
## Heliport Filters

### Description
Filters based on the HeLI-OTS method. Variants include Simple, Confidence, RawScore, and Probability filters.

### Parameters
- **languages** (list) - Required - Expected languages (ISO639 language codes)
- **thresholds** (float/list) - Required - Minimum identification score
- **topk** (int) - Optional - Number of top languages to consider (default 50, applies to RawScore and Probability filters)
```

--------------------------------

### Detokenize Parallel Texts

Source: https://helsinki-nlp.github.io/OpusFilter/preprocessors/detokenizer.html

Endpoint for detokenizing parallel text inputs using specified tokenizer configurations.

```APIDOC
## POST /detokenizer

### Description
Detokenize parallel texts based on provided tokenizer types and language codes.

### Method
POST

### Endpoint
/detokenizer

### Parameters
#### Request Body
- **tokenizer** (string or list) - Required - Tokenizer type or a list of types for each input.
- **languages** (list) - Required - A list of language codes for each input.
- **options** (dict or list) - Optional - Tokenizer options dictionary or a list of tokenizer dictionaries for multiple tokenizers.
```

--------------------------------

### LinguaFilter

Source: https://helsinki-nlp.github.io/OpusFilter/filters/script_and_language_identification_filters.html

Filter segments based on the Lingua method.

```APIDOC
## LinguaFilter

### Description
Filter segments based on the Lingua method.

### Parameters
- **languages** (list) - Required - Expected languages (ISO639 language codes) for the segments
- **thresholds** (float/list) - Required - Minimum identification confidence score for the segments
- **langid_languages** (list) - Optional - Limit detection to a list of ISO 639-1 codes (default null)
- **lingua_mode** (string) - Optional - Use lingua’s 'high' or 'low' accuracy mode (default 'low')
```

--------------------------------

### MonolingualSentenceSplitter

Source: https://helsinki-nlp.github.io/OpusFilter/preprocessors/monolingual_sentence_splitter.html

Splits monolingual text segments into sentences. This method is imported from the sentence-splitter library and uses a heuristic algorithm. It is primarily intended for monolingual data and raises an exception for parallel data by default.

```APIDOC
## MonolingualSentenceSplitter

### Description
Split monolingual text segments into sentences.

### Method
POST (assumed, as it processes input data)

### Endpoint
/split/sentences

### Parameters
#### Query Parameters
- **language** (string) - Required - language code for the input
- **non_breaking_prefix_file** (string) - Optional - override the language’s non-breaking prefix file by a custom one (default `null`)
- **enable_parallel** (boolean) - Optional - do not raise expection if the input is parallel data (default `false`)

### Request Body
- **text** (string) - Required - The monolingual text to be split into sentences.

### Request Example
{
  "text": "This is the first sentence. This is the second sentence."
}

### Response
#### Success Response (200)
- **sentences** (array of strings) - A list of sentences extracted from the input text.

#### Response Example
{
  "sentences": [
    "This is the first sentence.",
    "This is the second sentence."
  ]
}
```

--------------------------------

### RepetitionFilter

Source: https://helsinki-nlp.github.io/OpusFilter/filters/special_character_and_similarity_filters.html

Filters segments that contain repeated content. It identifies repetitions based on a specified threshold, minimum and maximum character length of the repeated sequence, and returns the number of repetitions found for the first match or zero if no significant repetitions are detected. Optional spaces between repetitions are ignored.

```APIDOC
## RepetitionFilter

### Description
Filter segments with repeated content. Useful e.g. for filtering data generated by a low-quality NMT model. The returned scores are the numbers of repetitions if at least threshold repetitions were found (first occurrence of the string is not counted), or zero if no repetitions were found, or all were below the threshold. The returned number of repetitions is for the first match, and it is possible that the segment contains longer repetitions.

There may be optional space character(s) between the repeated strings that are not counted to the length. The repeated string cannot start with a whitespace character but is not limited otherwise.

### Method
Not specified (likely part of a larger processing pipeline)

### Endpoint
Not applicable (filter within a processing pipeline)

### Parameters
- **threshold** (integer) - Number of repetitions required to activate the filter (optional, default 2).
- **min_length** (integer) - Minimum number of characters in the repeated sequence (optional, default 3).
- **max_length** (integer) - Maximum number of characters in the repeated sequence (optional, default 100).

### Request Example
Not applicable

### Response
- **repetitions** (integer) - The number of repetitions found for the first match, or zero if no repetitions were found or all were below the threshold.

### Response Example
{
  "repetitions": 5
}
```

--------------------------------

### Train Classifier

Source: https://helsinki-nlp.github.io/OpusFilter/functions/training_and_using_classifiers.html

Trains an sklearn classifier to produce a cleanness score for sentence pairs.

```APIDOC
## train_classifier

### Description
Train an `sklearn` classifier to produce a cleanness score for sentence pairs.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

#### Parameters
- **training_scores** (file) - Required - A file containing filter scores for training in JSON lines format produced with `score` function.
- **criterion** (string) - Required - Criterion to be used in classifier optimization (valid options are `CE`, `ROC_AUC`, `SSE`, `AIC` and `BIC`).
- **dev_scores** (file) - Optional - A file containing filter scores for training in JSON lines format produced with `score` function with and added item `label` added to each entry. `label` has value 1 for clean pairs and 0 for noisy pairs (optional; `dev_scores` is only used when the `criterion` is `ROC_AUC`).
- **model_type** (string) - Optional - Classifier model type selected from `sklearn` classifiers (default `LogisticRegression`).
- **model_parameters** (object) - Optional - Parameters for the `sklearn` classifier.
- **model** (file) - Required - Output model file.
- **features** (list) - Required - The features given to the classifier to be trained on, defined as a list of filter names.
  - **ExampleFilter** (object) - Optional - Configuration for a specific feature filter.
    - **clean-direction** (string) - Optional - The direction that indicates higher cleanness (valid options are `high` and `low`).
    - **quantiles** (object) - Optional - A dictionary specifying quantile boundaries for training example selection (default `{'min': 0, 'max': 1, 'initial': 0.1}`).

### Request Example
None

### Response
#### Success Response (200)
None

#### Response Example
None

### Notes
The classifier is optimized by training multiple classifier models with the training data divided differently into positive and negative examples based on the quantile boundaries specified in each feature. The model that achieves the highest criterion score is then saved in the output file.
```

--------------------------------

### Cite OpusFilter in BibTeX

Source: https://helsinki-nlp.github.io/OpusFilter/references.html

Use this BibTeX entry to cite the OpusFilter toolbox in academic publications.

```bibtex
@inproceedings{aulamo-etal-2020-opusfilter,
    title = "{O}pus{F}ilter: A Configurable Parallel Corpus Filtering Toolbox",
    author = {Aulamo, Mikko and Virpioja, Sami and Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = jul,
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-demos.20",
    doi = "10.18653/v1/2020.acl-demos.20",
    pages = "150--156"
}
```

--------------------------------

### SimilarityFilter

Source: https://helsinki-nlp.github.io/OpusFilter/filters/special_character_and_similarity_filters.html

Filters segments based on string or word sequence similarity using Levenshtein distance. It calculates normalized similarities and supports n-lingual input. Customizable parameters include threshold, weights for edit operations, unit of comparison, case sensitivity, and requirement for all pairs to meet the threshold.

```APIDOC
## SimilarityFilter

### Description
Filter segments based on string or word sequence similarity based on Levenshtein distance. The returned scores are normalized similarities (1 - edit distance / max edit distance) between the compared sequences for all language pairs. For n-lingual input, the scores will include C(n, 2) values.

### Method
Not specified (likely part of a larger processing pipeline)

### Endpoint
Not applicable (filter within a processing pipeline)

### Parameters
- **threshold** (float) - Filter segments if the similarity is equal or above the threshold (optional, default 0.9).
- **weights** (list of integers) - A list of three integers corresponding to the costs of three edit operations: insertion, deletion, substitution (optional; default `[1, 1, 1]`).
- **unit** (string) - Type of unit for calculating the distance (optional; `word` for words or any whitespace-separated units, and `character` or `char` for characters; the default is `char`).
- **lowercase** (boolean) - Lowercase strings as preprocessing (default `false`).
- **require_all** (boolean) - If True, all similarities (for pairs of n segments) have to be below the threshold; otherwise, at least one of the similarities has to be below the threshold.

### Request Example
Not applicable

### Response
- **scores** (list of floats) - Normalized similarities between the compared sequences for all language pairs. For n-lingual input, the scores will include C(n, 2) values.

### Response Example
{
  "scores": [0.92, 0.85, 0.98]
}
```

--------------------------------

### LengthRatioFilter

Source: https://helsinki-nlp.github.io/OpusFilter/filters/length_filters.html

Filters segments based on the ratio of their lengths. Segments are accepted if the ratio of the longer segment's length to the shorter segment's length is below the specified threshold.

```APIDOC
## LengthRatioFilter

### Description
Filtering based on ratio of the segment lengths.

### Parameters
#### Query Parameters
- **threshold** (float) - Required - Threshold for the length ratio
- **unit** (string) - Optional - Type of unit for calculating lengths (`word` or `character`; default `word`)

### Notes
Returned score is the higher length divided by the lower length, or infinity if either of the lengths are zero. In filtering, segment pairs are accepted if the ratio is below the given threshold. In order to use different units per language, the `unit` parameter can also be given as a list.
```

--------------------------------

### FastTextFilter

Source: https://helsinki-nlp.github.io/OpusFilter/filters/script_and_language_identification_filters.html

Filter segments based on Fasttext language identification models.

```APIDOC
## FastTextFilter

### Description
Filter segments based on the Fasttext language identification models. Requires installing optional libraries.

### Parameters
- **languages** (list) - Required - Expected languages (ISO639 language codes) for the segments
- **thresholds** (float/list) - Required - Minimum identification confidence score for the segments
- **model_path** (string) - Required - Path for a fasttext model
```

--------------------------------

### remove_duplicates

Source: https://helsinki-nlp.github.io/OpusFilter/functions/filtering_and_scoring.html

Filters out duplicate lines from parallel corpus files using hashing to manage memory efficiently for large files.

```APIDOC
## remove_duplicates

### Description
Filter out duplicate lines from parallel corpus files.

### Method
Not specified (assumed to be a command-line tool or function call)

### Endpoint
Not applicable

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

**Parameters:**
- `inputs`: input file(s)
- `outputs`: output file(s)
- `compare`: select files for duplicate comparison (optional; default `all` or a list of indices)
- `hash`: select hash algorithm from xxhash (optional; default `xxh64`)
- `overlap`: remove overlap with a second set of files (optional; default `null`)
- `letters_only`: remove all non-letters from intput strings before hashing (optional; default `false`)
- `letter_words_only`: remove words with non-letter characters before hashing (optional; default `false`)
- `lowercase`: lowercase input strings before hashing (optional; default `false`)
- `tokenizers`: load tokenizer specifications from a list (optional; use with `letter_words_only`)

### Request Example
```json
{
  "inputs": ["input1.txt", "input2.txt"],
  "outputs": ["output1.txt", "output2.txt"],
  "compare": [0],
  "lowercase": true
}
```

### Response
#### Success Response (200)
Output files containing filtered data.

#### Response Example
(No specific response body example provided, output is in files.)
```

--------------------------------

### filter

Source: https://helsinki-nlp.github.io/OpusFilter/functions/filtering_and_scoring.html

Filters parallel data using a combination of specified filters. Can either keep segments passing filters or reject segments failing at least one.

```APIDOC
## filter

### Description
Filter parallel data with a combination of filters.

### Method
Not specified (assumed to be a command-line tool or function call)

### Endpoint
Not applicable

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

**Parameters:**
- `inputs`: input files for segments to filter
- `outputs`: output files for filtered sentences
- `n_jobs`: number of sub processes to parallel run jobs. If not set, the default value is `default_n_jobs` in `common` section.
- `filters`: a list of filters to apply; see below
- `filterfalse`: yield segment pairs that do not pass at least one of the filters (optional; default `false`)

### Request Example
```json
{
  "inputs": ["input.txt"],
  "outputs": ["output.txt"],
  "filters": [
    {"LengthFilter": {"max_len": 100, "name": "max_length"}},
    {"LanguageIDFilter": {"source_lang": "en", "target_lang": "fr", "name": "lang_id"}}
  ],
  "filterfalse": false
}
```

### Response
#### Success Response (200)
Output files containing filtered sentences.

#### Response Example
(No specific response body example provided, output is in files.)
```

--------------------------------

### WordAlignFilter

Source: https://helsinki-nlp.github.io/OpusFilter/filters/alignment_model_filters.html

Filters segments based on word alignment scores using eflomal. A segment pair is accepted if scores for both directions are lower than the corresponding thresholds.

```APIDOC
## WordAlignFilter

### Description
Filter segments by word alignment scores using eflomal.

### Method
Not applicable (this describes a filter configuration, not an API endpoint).

### Endpoint
Not applicable.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
This filter is configured via parameters, not a request body.

### Parameters
- **src_threshold** (float) - Optional - Score threshold for the source language (default 0).
- **tgt_threshold** (float) - Optional - Score threshold for the target language (default 0).
- **src_tokenizer** (tuple) - Optional - Tokenizer for the source language (default `null`). Example: `[moses, en]`.
- **tgt_tokenizer** (tuple) - Optional - Tokenizer for the target language (default `null`). Example: `[moses, fr]`.
- **model** (int) - Optional - eflomal model type (default 3). 1: IBM1, 2: IBM1 + HMM, 3: IBM1 + HMM + fertility.
- **priors** (str) - Optional - Path to priors file for alignment (default `null`).
- **score_for_empty** (int) - Optional - Score value for empty input pairs (default -100).

### Request Example
```json
{
  "filter": "WordAlignFilter",
  "src_threshold": 0.8,
  "tgt_threshold": 0.7,
  "src_tokenizer": ["moses", "en"],
  "tgt_tokenizer": ["moses", "de"],
  "model": 3,
  "priors": "/path/to/priors.txt",
  "score_for_empty": -50
}
```

### Response
#### Success Response (200)
This filter does not return a response directly; it modifies the input data stream.

#### Response Example
Not applicable.

### Caveats
- `eflomal` is stochastic; results may vary between runs.
- Full replicability is not possible if this filter is used.
- Model parameters are estimated from input data, affecting results based on input size.
- Using `score` or `filter` with `filterfalse` on inputs larger than `chunksize` (default 100000) may yield suboptimal results.
- Parallel processing (`n_jobs` > 1) for `score` or `filter` can lead to different results regardless of other options.
```

--------------------------------

### NonZeroNumeralsFilter

Source: https://helsinki-nlp.github.io/OpusFilter/filters/special_character_and_similarity_filters.html

Filters segments based on a similarity measure of numerals (excluding zeros) between segments. It extracts non-zero numerals, preserves their order, and calculates a similarity score using Python's `difflib.SequenceMatcher.ratio()`. This filter supports n-lingual input.

```APIDOC
## NonZeroNumeralsFilter

### Description
Filter segments based on a similarity measure of numerals between the segments with zeros removed. Non-zero numerals are extracted from all segments preserving the relative order of the numerals. The similarity score between the numeral sequences is produced with `SequenceMatcher.ratio()` from Python’s `difflib` library.

### Method
Not specified (likely part of a larger processing pipeline)

### Endpoint
Not applicable (filter within a processing pipeline)

### Parameters
- **threshold** (float) - Minimum score threshold (default 0.5).
- **require_all** (boolean) - If True, all scores (for pairs of n segments) have to reach the threshold; otherwise, at least one ratio has to reach the threshold.

### Request Example
Not applicable

### Response
- **scores** (list of floats) - A list of similarity scores for all language pairs. For n-lingual input, the scores will include C(n, 2) values. In filtering, all pairwise scores have to equal to or be greater than the minimum threshold.

### Response Example
{
  "scores": [0.8, 0.7, 0.9]
}
```
Tokens:
Raw