### Install with test dependencies

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Install the package along with its test dependencies using pip.

```bash
pip install .[test]
```

--------------------------------

### Install optional dependencies

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Install the package with optional dependencies for full functionality, including test, eflomal, jieba, mecab, and varikn.

```bash
pip install .[test,eflomal,jieba,mecab,varikn]
```

--------------------------------

### Run filter with custom parameter options

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/usage.md

Example of performing a word length filter using individual command-line options instead of a single JSON object.

```bash
opusfilter-cmd filter --inputs wmt.fi.gz wmt.en.gz --outputs wmt_filtered.fi.gz wmt_filtered.en.gz --filters '[{"LengthFilter": {"unit": "word", "min_length": 1, "max_length": 100}}]'
```

--------------------------------

### Install OpusFilter via pip

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Install the package from PyPI with optional dependencies for specific features like sentence embeddings or tokenization.

```bash
# Basic installation
pip install opusfilter

# Install with all optional libraries (LASER, eflomal, jieba, mecab, etc.)
pip install opusfilter[all]

# Install specific extras
pip install opusfilter[laser]      # Sentence embeddings
pip install opusfilter[eflomal]    # Word alignment
pip install opusfilter[varikn]     # N-gram language models
pip install opusfilter[jieba]      # Chinese tokenization
pip install opusfilter[mecab]      # Japanese tokenization
```

--------------------------------

### Run filter with JSON parameters

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/usage.md

Example of performing a word length filter using a single JSON object for parameters.

```bash
opusfilter-cmd filter --parameters {"inputs": ["wmt.fi.gz", "wmt.en.gz"], "outputs": ["wmt_filtered.fi.gz", "wmt_filtered.en.gz"], "filters": [{"LengthFilter": {"unit": "word", "min_length": 1, "max_length": 100}}]}
```

--------------------------------

### slice

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/downloading_and_selecting_data.md

Extracts a slice of lines from input files based on start, stop, and step parameters.

```APIDOC
## slice

### Description
Take slice of lines from files.

### Parameters
* `inputs` (list of strings) - Required - A list of input files.
* `outputs` (list of strings) - Required - A list of output files.
* `start` (integer) - Optional - Start index (default 0).
* `stop` (integer) - Optional - Stop index (default `null`).
* `step` (integer) - Optional - Step size (default 1).

### Note
Either `start`, `stop`, or both of them should be given. If `stop` is not given, reads until the end of the file.
```

--------------------------------

### Define complete filtering pipeline

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

A comprehensive YAML configuration example for a parallel corpus filtering workflow, covering data acquisition, cleaning, model training, and splitting.

```yaml
# complete_pipeline.yaml
common:
  output_directory: output
  default_n_jobs: 4

steps:
  # 1. Download data
  - type: opus_read
    parameters:
      corpus_name: ParaCrawl
      source_language: fi
      target_language: en
      release: v4
      preprocessing: moses
      src_output: raw.fi.gz
      tgt_output: raw.en.gz

  # 2. Remove duplicates
  - type: remove_duplicates
    parameters:
      inputs: [raw.fi.gz, raw.en.gz]
      outputs: [dedup.fi.gz, dedup.en.gz]

  # 3. Initial filtering
  - type: filter
    parameters:
      inputs: [dedup.fi.gz, dedup.en.gz]
      outputs: [clean.fi.gz, clean.en.gz]
      filters:
        - LengthFilter:
            unit: word
            min_length: 1
            max_length: 100
        - LengthRatioFilter:
            threshold: 3
        - LongWordFilter:
            threshold: 40
        - HtmlTagFilter: {}
        - CharacterScoreFilter:
            scripts: [Latin, Latin]
            thresholds: [0.9, 0.9]
        - LangidFilter:
            languages: [fi, en]
            thresholds: [0.5, 0.5]

  # 4. Train language models
  - type: train_ngram
    parameters:
      data: clean.fi.gz
      model: fi.arpa.gz
      parameters:
        norder: 15
        dscale: 0.1

  - type: train_ngram
    parameters:
      data: clean.en.gz
      model: en.arpa.gz
      parameters:
        norder: 15
        dscale: 0.1

  # 5. Train alignment model
  - type: train_alignment
    parameters:
      src_data: clean.fi.gz
      tgt_data: clean.en.gz
      output: align.priors
      parameters:
        src_tokenizer: [moses, fi]
        tgt_tokenizer: [moses, en]

  # 6. Score all data
  - type: score
    parameters:
      inputs: [dedup.fi.gz, dedup.en.gz]
      output: scores.jsonl.gz
      filters:
        - CrossEntropyFilter:
            lm_params:
              - filename: fi.arpa.gz
              - filename: en.arpa.gz
        - WordAlignFilter:
            src_tokenizer: [moses, fi]
            tgt_tokenizer: [moses, en]
            priors: align.priors

  # 7. Final filtering with trained models
  - type: filter
    parameters:
      inputs: [dedup.fi.gz, dedup.en.gz]
      outputs: [final.fi.gz, final.en.gz]
      filters:
        - LengthFilter:
            min_length: 1
            max_length: 100
        - CrossEntropyFilter:
            thresholds: [50, 50]
            diff_threshold: 10
            lm_params:
              - filename: fi.arpa.gz
              - filename: en.arpa.gz
        - WordAlignFilter:
            src_threshold: -5
            tgt_threshold: -5
            src_tokenizer: [moses, fi]
            tgt_tokenizer: [moses, en]
            priors: align.priors

  # 8. Split into train/dev/test
  - type: split
    parameters:
      inputs: [final.fi.gz, final.en.gz]
      outputs: [test.fi.gz, test.en.gz]
      outputs_2: [traindev.fi.gz, traindev.en.gz]
      divisor: 100
      threshold: 1

  - type: split
    parameters:
      inputs: [traindev.fi.gz, traindev.en.gz]
      outputs: [dev.fi.gz, dev.en.gz]
      outputs_2: [train.fi.gz, train.en.gz]
      divisor: 99
      threshold: 1
```

--------------------------------

### Define Basic Pipeline Configuration

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Configure common settings and processing steps using a YAML file structure.

```yaml
# config.yaml - Basic pipeline structure
common:
  output_directory: my_output
  chunksize: 100000           # Chunk size for filter/score steps
  default_n_jobs: 4           # Parallel processes for filter/score/preprocess
  constants:
    source: en
    target: fi

steps:
  - type: opus_read
    parameters:
      corpus_name: ParaCrawl
      source_language: !var source
      target_language: !var target
      release: v4
      preprocessing: raw
      src_output: !varstr "paracrawl.{source}.gz"
      tgt_output: !varstr "paracrawl.{target}.gz"

  - type: filter
    parameters:
      inputs:
        - !varstr "paracrawl.{source}.gz"
        - !varstr "paracrawl.{target}.gz"
      outputs:
        - !varstr "filtered.{source}.gz"
        - !varstr "filtered.{target}.gz"
      filters:
        - LengthFilter:
            unit: word
            min_length: 1
            max_length: 100
```

--------------------------------

### Generate pipeline diagram

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Creates a visual representation of the pipeline defined in the configuration file.

```bash
opusfilter-diagram config.yaml pipeline.png
```

--------------------------------

### head

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/downloading_and_selecting_data.md

Extracts the first n lines from input files and writes them to corresponding output files.

```APIDOC
## head

### Description
Take the first n lines from files.

### Parameters
* `inputs` (list of strings) - Required - A list of input files.
* `outputs` (list of strings) - Required - A list of output files.
* `n` (integer) - Required - Number of output lines.
```

--------------------------------

### Run Full OpusFilter Pipeline

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Execute a complete OpusFilter pipeline using a configuration file. This is the primary command for processing data.

```bash
opusfilter config.yaml
```

--------------------------------

### Run full linting with pylint

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Perform a comprehensive linting check on the 'opusfilter/' directory using pylint. Recommended for contributions.

```bash
pylint opusfilter/
```

--------------------------------

### Reuse filter definitions with YAML anchors

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/usage.md

Demonstrates using YAML anchors (&) and references (*) to apply the same filter configuration to multiple datasets.

```yaml
  - type: filter
    parameters:
      inputs:
      - paracrawl.fi.gz
      - paracrawl.en.gz
      outputs:
      - paracrawl_filtered.fi.gz
      - paracrawl_filtered.en.gz
      filters: &myfilters
        - LengthFilter:
            unit: word
            min_length: 1
            max_length: 100

        - LengthRatioFilter:
            unit: word
            threshold: 3

  - type: filter
    parameters:
      inputs:
      - wmt.fi.gz
      - wmt.en.gz
      outputs:
      - wmt_filtered.fi.gz
      - wmt_filtered.en.gz
      filters: *myfilters
```

--------------------------------

### Run OpusFilter with Options

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Run an OpusFilter pipeline with command-line options to control overwriting, processing specific steps, and parallel job execution.

```bash
opusfilter --overwrite --last 5 --n-jobs 8 config.yaml
```

```bash
opusfilter --single 3 config.yaml  # Run only step 3
```

--------------------------------

### Using Variables and Constants in OpusFilter

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Demonstrates using YAML variables and constants for reusable configurations in OpusFilter pipelines. Variables can be used for languages, file paths, and more.

```yaml
common:
  constants:
    src: en
    tgt: fi

steps:
  # Using variables in multiple steps
  - type: opus_read
    parameters:
      corpus_name: ParaCrawl
      source_language: !var src
      target_language: !var tgt
      release: v4
      preprocessing: raw
      src_output: !varstr "paracrawl.{src}.gz"
      tgt_output: !varstr "paracrawl.{tgt}.gz"

  # Expand step for multiple language pairs
  - type: filter
    parameters:
      inputs:
        - !varstr "paracrawl.{src}-{tgt}.{src}.gz"
        - !varstr "paracrawl.{src}-{tgt}.{tgt}.gz"
      outputs:
        - !varstr "filtered.{src}-{tgt}.{src}.gz"
        - !varstr "filtered.{src}-{tgt}.{tgt}.gz"
      filters: &myfilters
        - LengthFilter:
            min_length: 1
            max_length: 100
    variables:
      tgt: [fi, sv, de]           # Creates 3 steps

  # Reuse filter definitions with YAML anchors
  - type: filter
    parameters:
      inputs: [other.fi.gz, other.en.gz]
      outputs: [other_filtered.fi.gz, other_filtered.en.gz]
      filters: *myfilters
```

--------------------------------

### Generate OpusFilter configuration diagrams

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/command_line_tools.md

Creates a directed acyclic graph from a configuration file using Graphviz. Output format is determined by the file extension.

```bash
opusfilter-diagram [--rankdir {TB,LR}] FILE FILE
```

--------------------------------

### Run flake8 with statistics

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Execute flake8 with options to count errors and display statistics.

```bash
flake8 opusfilter --count --exit-zero --statistics
```

--------------------------------

### Execute OpusFilter command syntax

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/usage.md

General syntax for running the opusfilter-cmd script.

```bash
opusfilter-cmd [--overwrite] [--outputdir OUTDIR] FUNCTION [--parameters PARAMETERS] [PARAMETER-OPTION] ...
```

--------------------------------

### Head, Tail, Slice - Select Line Ranges

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Extracts specific line ranges from files using head, tail, or slice operations.

```yaml
steps:
  # First 10000 lines
  - type: head
    parameters:
      inputs: [corpus.fi.gz, corpus.en.gz]
      outputs: [first_10k.fi.gz, first_10k.en.gz]
      n: 10000

  # Last 1000 lines
  - type: tail
    parameters:
      inputs: [corpus.fi.gz, corpus.en.gz]
      outputs: [last_1k.fi.gz, last_1k.en.gz]
      n: 1000

  # Lines 1000-2000 (Python slice notation)
  - type: slice
    parameters:
      inputs: [corpus.fi.gz, corpus.en.gz]
      outputs: [slice.fi.gz, slice.en.gz]
      start: 1000
      stop: 2000
      step: 1
```

--------------------------------

### Run OpusFilter command

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/usage.md

Executes the main opusfilter script using a specified YAML configuration file.

```bash
opusfilter [--overwrite] [--last LAST] [--single SINGLE] [--n-jobs N_JOBS] CONFIG
```

--------------------------------

### Split Data into Parts

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Split files deterministically using hash-based methods. Configure inputs, multiple output pairs, divisor, and a seed for reproducibility.

```yaml
steps:
  # Split 90/10 for train/test
  - type: split
    parameters:
      inputs: [corpus.fi.gz, corpus.en.gz]
      outputs: [train.fi.gz, train.en.gz]
      outputs_2: [test.fi.gz, test.en.gz]
      divisor: 10                 # 1/10 to outputs, 9/10 to outputs_2
      threshold: 1
      compare: [0]                # Base decision on source only
      seed: 42

  # Split into equal halves
  - type: split
    parameters:
      inputs: [corpus.fi.gz, corpus.en.gz]
      outputs: [part1.fi.gz, part1.en.gz]
      outputs_2: [part2.fi.gz, part2.en.gz]
      divisor: 2
      threshold: 1
```

--------------------------------

### RegExpFilter Configuration

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/filters/special_character_and_similarity_filters.md

This section details how to configure the RegExpFilter using its parameters.

```APIDOC
## RegExpFilter

Filter out segments that match (or do not match) a arbitrary regular expression.

### Parameters

#### Request Body

- **regexps** (string or list of strings) - Required - A regular expression or a list of expressions to match.
- **accept_match** (boolean) - Optional - Accept matching segments instead of rejecting (default `false`).

### Description

You can either provide a single regexp or one for each language in the parallel data. If `accept_match` is `false`, the pair is accepted only if none of the segment match the corresponding regular experssion. If `accept_match` is `true`, the pair is accepted only if all segments match the corresponding regular expression.

The [regex](https://pypi.org/project/regex/) module is used for the regular expressions.
```

--------------------------------

### RegExpSub Configuration

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/preprocessors/reg_exp_sub.md

Details on how to configure the RegExpSub filter using patterns and language-specific overrides.

```APIDOC
## RegExpSub Configuration

### Description
Run a sequence of arbitrary regular expression substitutions on input text. Substitutions are applied in the order provided.

### Parameters
- **patterns** (list) - Required - A list of default substitution patterns.
- **lang_patterns** (dict/list) - Optional - A dictionary or list of patterns to use for specific languages, overriding default patterns at the corresponding index.

### Pattern Structure
Each pattern is a 4-tuple:
- **regex** (string) - The regular expression to match.
- **replacement** (string) - The replacement string.
- **count** (int) - Number of substitutions (0 = substitute all).
- **flags** (list) - List of re library flag constants (e.g., ["I", "A"]).

### Usage Example
{
  "patterns": [["\\s+", " ", 0, []]],
  "lang_patterns": {
    "en": [["\\d+", "#", 0, []]]
  }
}
```

--------------------------------

### Detokenizer Configuration

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/preprocessors/detokenizer.md

Defines the parameters required to initialize and use the Detokenizer for parallel text processing.

```APIDOC
## Detokenizer

### Description
Detokenize parallel texts using specified tokenizer configurations.

### Parameters
- **tokenizer** (string or list) - Required - Tokenizer type or a list of types for each input.
- **languages** (list) - Required - A list of language codes for each input.
- **options** (dict or list) - Optional - Tokenizer options dictionary or a list of tokenizer dictionaries for multiple tokenizers.
```

--------------------------------

### Join Score Files Configuration

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Configure the join step to combine multiple score files into a single output. Specify input files and the keys for merging, which can be at the root level or nested.

```yaml
steps:
  - type: join
    parameters:
      inputs:
        - scores.jsonl.gz
        - external_scores.src
        - external_scores.tgt
      keys:
        - null                    # Merge at root level
        - ExternalScore.src       # Nest under ExternalScore.src
        - ExternalScore.tgt
      output: joined_scores.jsonl.gz
```

--------------------------------

### download

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/downloading_and_selecting_data.md

Downloads a file from a given URL to a specified output file.

```APIDOC
## download

### Description
Download a file from URL.

### Parameters
* `url` (string) - Required - URL for the file to download.
* `output` (string) - Required - Output file.
```

--------------------------------

### product

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/downloading_and_selecting_data.md

Creates a Cartesian product of parallel segments from input files with optional sampling.

```APIDOC
## product

### Description
Create a Cartesian product of parallel segments and optionally sample from them.

### Parameters
- **inputs** (list) - Required - A list of input files lists
- **outputs** (list) - Required - A list of output files
- **skip_empty** (boolean) - Optional - Skip empty lines (default: true)
- **skip_duplicates** (boolean) - Optional - Skip duplicate lines per language (default: true)
- **k** (integer) - Optional - Sample at most k random items per product
- **seed** (integer) - Optional - Seed for the random generator
```

--------------------------------

### Define Variables for Step Expansion

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/usage.md

Use the variables key to define lists of values, causing the step to expand into multiple substeps for each provided value.

```yaml
common:
  constants:
    source: en

steps:
  - type: concatenate
    parameters:
      inputs:
      - !varstr "file1.{source}-{target}.gz"
      - !varstr "file2.{source}-{target}.gz"
      output: !varstr "all.{source}-{target}.gz"
    variables:
      target: [fi, sv]
```

--------------------------------

### Sort imports with isort

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Automatically sort imports in the specified directories using isort.

```bash
isort opusfilter/ opusfilter/cli/
```

--------------------------------

### Run all tests with pytest

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Execute all tests in the 'tests/' directory using pytest.

```bash
pytest tests/
```

--------------------------------

### Standard Python import organization

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Organize imports into standard library, third-party, and local modules. Ensure proper ordering and separation.

```python
# Standard library imports
import os
import logging
from typing import Iterator, List, Tuple

# Third-party imports
import numpy as np
import pandas as pd
import regex

# Local imports
from . import FilterABC, ConfigurationError
from .util import file_open, count_lines
```

--------------------------------

### Filter implementation pattern

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

A template for implementing new filters, including score direction, thresholds, initialization, scoring, and acceptance logic.

```python
class NewFilter(FilterABC):
    """Brief description of the filter"""

    score_direction = CLEAN_LOW  # or CLEAN_HIGH, CLEAN_BETWEEN, etc.
    accept_threshold = <value>
    reject_threshold = <value>

    def __init__(self, required_param, optional_param=default, **kwargs):
        # Validate parameters
        self.required_param = required_param
        self.optional_param = optional_param
        super().__init__(**kwargs)

    def score(self, pairs):
        """Yield scores for each sentence pair"""
        for pair in pairs:
            # Calculate score
            yield score

    def accept(self, score):
        """Return True if score passes the filter"""
        return <condition>
```

--------------------------------

### write

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/downloading_and_selecting_data.md

Writes a specified string into a file.

```APIDOC
## write

### Description
Write a specified string into a file.

### Parameters
- **output** (string) - Required - Output file
- **data** (string) - Required - Input data to write to the output
```

--------------------------------

### Implement a Custom Preprocessor

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/preprocessors/custom_preprocessors.md

Define a custom preprocessor by inheriting from PreprocessorABC and implementing the abstract process method. The process method should be a generator yielding modified segments.

```python
from opusfilter.dictionary import Dictionary
from opusfilter.preprocessor import PreprocessorABC

class MyPreprocessor(PreprocessorABC):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        # Add any custom initialization here

    def process(self, segments, f_idx=None):
        # Example: Simple segment modification
        for segment in segments:
            modified_segment = segment.strip() + " [processed]"
            yield modified_segment

```

--------------------------------

### Preprocess - Text Preprocessing

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Applies various text transformations including normalization, tokenization, regex substitution, and subword segmentation.

```yaml
steps:
  - type: preprocess
    parameters:
      inputs: [raw.fi.gz, raw.en.gz]
      outputs: [processed.fi.gz, processed.en.gz]
      n_jobs: 4
      preprocessors:
        # Normalize whitespace
        - WhitespaceNormalizer: {}

        # Tokenize text
        - Tokenizer:
            tokenizer: moses
            languages: [fi, en]

        # Apply regex substitutions
        - RegExpSub:
            patterns:
              - ["\\s+", " ", 0, []]           # Collapse whitespace
              - ["^\\s+|\\s+$", "", 0, []]     # Trim
            lang_patterns:
              0:                                # Finnish-specific
                - ["ä", "a", 0, ["I"]]

        # Subword segmentation
        - BPESegmentation:
            model: bpe.model
            dropout: 0.1

        - MorfessorSegmentation:
            model: morfessor.bin
```

--------------------------------

### Preprocess Text

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/preprocessing_text.md

Applies a combination of preprocessors to input text files and saves the results to output files.

```APIDOC
## preprocess

### Description
Preprocess text with a combination of preprocessors.

### Method
POST

### Endpoint
/preprocess

### Parameters
#### Query Parameters
- **inputs** (list of str) - Required - input files for segments to preprocess
- **outputs** (list of str) - Required - output files for preprocessed segments
- **n_jobs** (int) - Optional - number of sub processes to parallel run jobs. If not set, the default value is `default_n_jobs` in `common` section.

#### Request Body
- **preprocessors** (list of dict) - Required - a list of preprocessors to apply; see below
  The preprocessors parameter is a list of dictionaries, each representing one preprocessor. The top level should typically include a single key that defines the class name for the preprocessor (e.g. `WhitespaceNormalizer`). Additionally it can include a special key `module` for defining module name for custom preprocessors.
  Under the class name there is a dictionary the defines the parameters of the preprocessors. The are mostly specific to the preprocessor class; see [Available preprocessors](preprocessors) for ready-made preprocessors.

### Request Example
```json
{
  "inputs": ["input1.txt", "input2.txt"],
  "outputs": ["output1.txt", "output2.txt"],
  "n_jobs": 4,
  "preprocessors": [
    {
      "WhitespaceNormalizer": {}
    },
    {
      "module": "my_custom_preprocessors",
      "MyCustomPreprocessor": {
        "param1": "value1"
      }
    }
  ]
}
```

### Response
#### Success Response (200)
- **status** (str) - Indicates the success of the operation.

#### Response Example
```json
{
  "status": "Preprocessing complete."
}
```
```

--------------------------------

### Sort Files Configuration

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Configure the sort step to sort parallel files based on score values. Specify input/output files, the score file, the key for sorting, and the sort order.

```yaml
steps:
  - type: sort
    parameters:
      inputs: [corpus.fi.gz, corpus.en.gz]
      outputs: [sorted.fi.gz, sorted.en.gz]
      values: scores.jsonl.gz
      key: CrossEntropyFilter.0   # Dot notation for nested keys
      reverse: false              # Ascending order
      type: float
```

--------------------------------

### Concatenate Multiple Text Files

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Combine several input files into a single output file using the concatenate step.

```yaml
steps:
  - type: concatenate
    parameters:
      inputs:
        - paracrawl.fi.gz
        - wmt.fi.gz
        - europarl.fi.gz
      output: all_data.fi.gz

  - type: concatenate
    parameters:
      inputs:
        - paracrawl.en.gz
        - wmt.en.gz
        - europarl.en.gz
      output: all_data.en.gz
```

--------------------------------

### Download Corpora with opus_read

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Use the opus_read step to fetch parallel corpora from the OPUS collection.

```yaml
steps:
  - type: opus_read
    parameters:
      corpus_name: ParaCrawl      # Corpus name in OPUS
      source_language: fi         # Source language code
      target_language: en         # Target language code
      release: v4                 # Corpus version
      preprocessing: raw          # 'raw', 'moses', or 'xml'
      src_output: paracrawl.fi.gz
      tgt_output: paracrawl.en.gz
      suppress_prompts: true      # Skip download confirmation

  # Download multiple corpora
  - type: opus_read
    parameters:
      corpus_name: WMT-News
      source_language: fi
      target_language: en
      release: v2019
      preprocessing: moses        # Recommended when available
      src_output: wmt.fi.gz
      tgt_output: wmt.en.gz
```

--------------------------------

### Run a single test file with pytest

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Execute tests within a specific file in the 'tests/' directory.

```bash
pytest tests/test_filename.py
```

--------------------------------

### Analyze Scores with opusfilter-scores

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Utilize opusfilter-scores for various analyses of score files, including listing, describing, and visualizing score distributions and correlations.

```bash
opusfilter-scores list scores.jsonl.gz
```

```bash
opusfilter-scores describe scores.jsonl.gz
```

```bash
opusfilter-scores hist scores.jsonl.gz output.png
```

```bash
opusfilter-scores corr scores.jsonl.gz correlation.png
```

--------------------------------

### Run OpusFilter Command Without Config File

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Execute a single OpusFilter command, such as 'filter', directly from the command line without a configuration file. Specify inputs, outputs, and filters as arguments.

```bash
opusfilter-cmd filter \
  --inputs corpus.fi.gz corpus.en.gz \
  --outputs filtered.fi.gz filtered.en.gz \
  --filters '[{"LengthFilter": {"unit": "word", "min_length": 1, "max_length": 100}}]'
```

--------------------------------

### Format code with black

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Automatically format Python code in the specified directories using the black formatter.

```bash
black opusfilter/ opusfilter/cli/
```

--------------------------------

### Train Classifier Configuration

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Configure the train_classifier step to train a scikit-learn classifier for segment pair cleanness scores. Requires training and development scores, and allows feature customization.

```yaml
steps:
  - type: train_classifier
    parameters:
      training_scores: scores.jsonl.gz
      dev_scores: dev_scores_with_labels.jsonl.gz  # Required for ROC_AUC
      model: classifier.pkl
      criterion: ROC_AUC          # 'CE', 'ROC_AUC', 'SSE', 'AIC', 'BIC'
      model_type: LogisticRegression
      model_parameters: {}
      features:
        LengthRatioFilter:
          clean-direction: low
          quantiles:
            min: 0
            max: 1
            initial: 0.1
        LangidFilter:
          clean-direction: high
          quantiles:
            min: 0
            max: 1
            initial: 0.1

  - type: classify
    parameters:
      model: classifier.pkl
      scores: all_scores.jsonl.gz
      output_probabilities: probabilities.txt
      output_labels: labels.txt
```

--------------------------------

### tail

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/downloading_and_selecting_data.md

Extracts the last n lines from input files and writes them to corresponding output files. Note: Memory usage is proportional to n.

```APIDOC
## tail

### Description
Take the last n lines from files.

### Parameters
* `inputs` (list of strings) - Required - A list of input files.
* `outputs` (list of strings) - Required - A list of output files.
* `n` (integer) - Required - Number of output lines.

### Note
The memory requirement of `tail` is proportional to n. Use `slice` if you need all except the first n lines.
```

--------------------------------

### split

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/downloading_and_selecting_data.md

Splits files into two parts based on a hash operation and a divisor. Useful for deterministic content-based splitting.

```APIDOC
## split

### Description
Split files to two parts giving the approximative proportions as fractions.

### Parameters
* `inputs` (list of strings) - Required - Input file(s).
* `outputs` (list of strings) - Required - Output file(s) for selected lines.
* `outputs_2` (list of strings) - Optional - Output file(s) for the rest of the lines.
* `divisor` (integer) - Required - Divisor for the modulo operation (e.g. 2 for splitting to equal sized parts).
* `threshold` (integer) - Optional - Threshold for the output of the modulo operation (default 1).
* `compare` (list of integers or string 'all') - Optional - Select files to use for hash operation (default 'all' or a list of indices).
* `hash` (string) - Optional - Select hash algorithm from xxhash (default 'xxh64').
* `seed` (integer) - Optional - Integer seed for the hash algorithm (default 0).

### Note
Input files are processed line by line in parallel. The decision is deterministic and based only on the content of the lines. The `compare` parameter can be used to select which input files are used to generate the content for the hash function. For example, if you have source and target language files, and you want that the split depends only on the source or target sentence, set `compare` to `[0]` or `[1]`, respectively. The divisors used in consecutive splits should not themselves have common divisors, or the proportion of the data in the output files may be unexpected. Distinct prime numbers are good choices. Also setting a different `seed` value for the hash functions prevents the issue.
```

--------------------------------

### Join Score Files

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/using_score_files.md

Combines two or more score files into a single output file. Scores can be nested under specified keys.

```APIDOC
## join

### Description
Join two or more score files.

### Parameters
#### Path Parameters
* `inputs` (list) - Required - input files containing scores in JSON Lines format
* `output` (string) - Required - output file for joined scores
* `keys` (list) - Optional - a list containing dictionary keys for each input file (default `null`)

### Request Example
```json
{
  "type": "join",
  "parameters": {
    "inputs": [
      "scores.jsonl.gz",
      "myscores.src",
      "myscores.tgt"
    ],
    "keys": [
      null,
      "MyScore.src",
      "MyScore.tgt"
    ],
    "output": "scores-joined.jsonl.gz"
  }
}
```

### Response
(No specific success response details provided for this operation, but the output file will contain joined scores.)
```

--------------------------------

### Generate Automatic OpusFilter Configuration

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Use opusfilter-autogen to automatically create a configuration file based on input files, languages, scripts, and analysis methods. Options include plotting and sample size.

```bash
opusfilter-autogen \
  --files corpus.fi.gz corpus.en.gz \
  --langs fi en \
  --scripts Latin Latin \
  --method clustering \
  --sample-size 100000 \
  --plot \
  -o auto_config.yaml
```

--------------------------------

### Check code style with flake8

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Perform a style check on the 'opusfilter/' directory using flake8.

```bash
flake8 opusfilter/
```

--------------------------------

### Apply Filters to Parallel Data

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Configure various filters to process parallel data. Specify input/output files, parallel processing jobs, and a list of filters with their parameters.

```yaml
steps:
  - type: filter
    parameters:
      inputs: [all.fi.gz, all.en.gz]
      outputs: [filtered.fi.gz, filtered.en.gz]
      n_jobs: 4                   # Parallel processing
      filterfalse: false          # Set true to output rejected pairs
      filters:
        # Length-based filters
        - LengthFilter:
            unit: word            # 'word' or 'character'/'char'
            min_length: 1
            max_length: 100
            pass_empty: false

        - LengthRatioFilter:
            unit: word
            threshold: 3          # Max ratio of longer/shorter

        - LongWordFilter:
            threshold: 40         # Max word length

        - AverageWordLengthFilter:
            min_length: 2
            max_length: 20

        # Script and language filters
        - CharacterScoreFilter:
            scripts: [Latin, Latin]
            thresholds: [0.9, 0.9]

        - AlphabetRatioFilter:
            threshold: 0.75
            exclude_whitespace: false

        - LangidFilter:
            languages: [fi, en]
            thresholds: [0.5, 0.5]
            langid_languages: [fi, en, sv, de]  # Limit detection

        - LinguaFilter:
            languages: [fi, en]
            thresholds: [0.5, 0.5]
            lingua_mode: low      # 'low' or 'high' accuracy

        # Content filters
        - HtmlTagFilter: {}

        - TerminalPunctuationFilter:
            threshold: -2

        - NonZeroNumeralsFilter:
            threshold: 0.5

        - RepetitionFilter:
            threshold: 2
            min_length: 3
            max_length: 100

        - RegExpFilter:
            regexps: ["^https?://"]
            accept_match: false

        - SimilarityFilter:
            threshold: 0.9
            unit: char
            lowercase: true
```

--------------------------------

### Download parallel corpus with opus_read

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/usage.md

A basic configuration to download a specific corpus release from OPUS and save the segments to compressed files.

```yaml
steps:
  - type: opus_read
    parameters:
      corpus_name: ParaCrawl
      source_language: fi
      target_language: en
      release: v4
      preprocessing: raw
      src_output: paracrawl.fi.gz
      tgt_output: paracrawl.en.gz
```

--------------------------------

### concatenate

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/downloading_and_selecting_data.md

Concatenates two or more text files into a single output file.

```APIDOC
## concatenate

### Description
Concatenate two or more text files.

### Parameters
* `inputs` (list of strings) - Required - A list of input files.
* `output` (string) - Required - Output file.
```

--------------------------------

### Train_ngram - Train Language Models

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Trains character-based n-gram language models using VariKN for filtering purposes.

```yaml
steps:
  # Train background model from concatenated data
  - type: concatenate
    parameters:
      inputs: [corpus.fi.gz, corpus.en.gz]
      output: concatenated.gz

  - type: train_ngram
    parameters:
      data: concatenated.gz
      model: background.arpa.gz
      parameters:
        norder: 5                 # Max n-gram order (0 = no limit)
        dscale: 1.0               # Model size scale (smaller = larger model)
        arpa: true                # Output ARPA format
        cutoffs: "0 0 1"          # Pruning cutoffs

  # Train language-specific models
  - type: train_ngram
    parameters:
      data: clean.fi.gz
      model: fi.arpa.gz
      parameters:
        norder: 15
        dscale: 0.1
        segmentation:
          type: char              # 'char', 'none', 'bpe', 'morfessor'
```

--------------------------------

### train_bpe

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/training_language_and_alignment_models.md

Trains a subword segmentation model using Byte Pair Encoding (BPE).

```APIDOC
## train_bpe

### Description
Trains a subword segmentation model with BPE as described by Sennrich et al. (2016).

### Parameters
- **input** (string) - Required - Input file name for training data
- **model** (string) - Required - Output file name for the model
- **symbols** (int) - Optional - Create this many new symbols (default 10000)
- **min_frequency** (int) - Optional - Stop if no symbol pair has frequency equal or above the threshold (default 2)
- **num_workers** (int) - Optional - Number of processors to process texts (default 1)
```

--------------------------------

### Run pytest with verbose output

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Execute all tests with verbose output enabled to see more details.

```bash
pytest -v tests/
```

--------------------------------

### Define Constants in OpusFilter Configuration

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/usage.md

Use the constants key to define variables with global or local scope, enabling dynamic file path generation.

```yaml
common:
  constants:
    source: en

steps:
  - type: concatenate
    parameters:
      inputs:
      - !varstr "file1.{source}-{target}.gz"
      - !varstr "file2.{source}-{target}.gz"
      output: !varstr "all.{source}-{target}.gz"
    constants:
      target: fi
```

--------------------------------

### Configure Custom Uppercase Filter in YAML

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/filters/custom_filters.md

This YAML configuration demonstrates how to select and configure the custom `UppercaseFilter` within Opusfilter. It specifies the filter type, parameters including the threshold, and the module where the custom filter is defined.

```yaml
steps:

  ...

  - type: filter
    parameters:

      ...

      filters:

        - UppercaseFilter:
            threshold: 0.5
          module: customfilter
```

--------------------------------

### train_ngram

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/training_language_and_alignment_models.md

Trains a character-based varigram language model using VariKN. This model can be utilized by CrossEntropyFilter and CrossEntropyDifferenceFilter.

```APIDOC
## train_ngram

### Description
Train a character-based varigram language model with VariKN. Can be used for [`CrossEntropyFilter`](CrossEntropyFilter) and [`CrossEntropyDifferenceFilter`](CrossEntropyDifferenceFilter).

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* `data` (string) - Required - input file name for training data
* `model` (string) - Required - output file name for the model
* `parameters` (object) - Optional - training options for VariKN and tokenization
  * `optdata` (string) - Optional - filename for optimization data (default: `""` = use leave-one-out estimation instead)
  * `norder` (integer) - Optional - limit model order (default: `0` = no limit)
  * `dscale` (float) - Optional - model size scale factor (default: `0.001`; smaller value gives a larger model)
  * `dscale2` (float) - Optional - model size scaling during pruning step (default: `0` = no pruning)
  * `arpa` (boolean) - Optional - output ARPA instead of binary LM (default: `true`)
  * `use_3nzer` (boolean) - Optional - use 3 discounts per order instead of one (default: `false`)
  * `absolute` (boolean) - Optional - use absolute discounting instead of Kneser-Ney smoothing (default: `false`)
  * `cutoffs` (string) - Optional - use the specified cutoffs (default: `"0 0 1"`; the last value is used for all higher order n-grams)
  * `segmentation` (object) - Optional - subword segmentation options (default: `{}`)
    * `type` (string) - Required - defines the subword segmentation type (e.g., `char`, `none`, `bpe`, `morfessor`)
    * `model` (string) - Required for `bpe` and `morfessor` - file for a trained segmentation model
    * Additional parameters are passed as options for the specified model.
  * `mb` (string) - Optional - word-internal boundary marking (default: `""`; e.g., `"^#"` or `"@@$"`) 
  * `wb` (string) - Optional - word boundary tag (default: `"<w>"`)

### Request Example
```json
{
  "data": "train.txt",
  "model": "model.lm",
  "parameters": {
    "dscale": 0.0005,
    "segmentation": {
      "type": "bpe",
      "model": "bpe.model"
    }
  }
}
```

### Response
#### Success Response (200)
This command does not return a JSON response. It creates model files on disk.

#### Response Example
N/A
```

--------------------------------

### Join score files with OpusFilter

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/docs/functions/using_score_files.md

Use the join operation to merge multiple score files into a single JSONL output. Keys can be nested using dot notation to structure the resulting JSON objects.

```yaml
  - type: join
    parameters:
      inputs:
      - scores.jsonl.gz
      - myscores.src
      - myscores.tgt
      keys:
      - null
      - MyScore.src
      - MyScore.tgt
      output: scores-joined.jsonl.gz
```

--------------------------------

### Perform type checking with mypy

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Check for type errors in the specified directories using mypy.

```bash
mypy opusfilter/ opusfilter/cli/
```

--------------------------------

### Train_alignment - Train Word Alignment Priors

Source: https://context7.com/helsinki-nlp/opusfilter/llms.txt

Trains word alignment priors using eflomal for downstream alignment-based filtering.

```yaml
steps:
  - type: train_alignment
    parameters:
      src_data: clean.fi.gz
      tgt_data: clean.en.gz
      output: alignment.priors
      scores: alignment_scores.txt  # Optional: save training scores
      parameters:
        src_tokenizer: [moses, fi]
        tgt_tokenizer: [moses, en]
        model: 3                    # 1=IBM1, 2=IBM1+HMM, 3=IBM1+HMM+fertility
```

--------------------------------

### Iterate over sentence pairs

Source: https://github.com/helsinki-nlp/opusfilter/blob/develop/AGENTS.md

Standard loop structure for processing pairs of source and target strings.

```python
for pair in pairs:
    # pair is a list/tuple of strings
    # pair[0] is source, pair[1] is target, etc.
```