OpenKiwi

https://unbabel.github.io/OpenKiwi/
Admin
OpenKiwi is a Pytorch-based open-source framework that implements state-of-the-art machine...
Tokens:187,776
Snippets:1,560
Trust Score:9.7
Update:4 months ago
Show doc for...
Tokens:
Raw
### Generate Example Configuration File

Source: https://unbabel.github.io/OpenKiwi/usage.html

Use this command to print an example configuration file to the terminal, which can then be redirected to a file for customization.

```bash
kiwi (train|pretrain|predict|evaluate|search) --example
```

--------------------------------

### Redirect Example Configuration to File

Source: https://unbabel.github.io/OpenKiwi/usage.html

Saves the example configuration for the 'train' pipeline to a YAML file named 'train.yaml'. This is useful for creating custom configuration files.

```bash
kiwi train --example > train.yaml
```

--------------------------------

### Test Step Implementation Example (Single Dataloader)

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/qe_system/index.html

A comprehensive example of a test_step implementation for a single dataloader, including data unpacking, model inference, loss calculation, logging example images, and accuracy calculation.

```python
# CASE 1: A single test dataset
def test_step(self, batch, batch_idx):
    x, y = batch

    # implement your own
    out = self(x)
    loss = self.loss(out, y)

    # log 6 example images
    # or generated text... or whatever
    sample_imgs = x[:6]
    grid = torchvision.utils.make_grid(sample_imgs)
    self.logger.experiment.add_image('example_images', grid, 0)

    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)

    # all optional...
    # return whatever you need for the collation function test_epoch_end
    output = OrderedDict({
        'val_loss': loss_val,
        'val_acc': torch.tensor(val_acc), # everything must be a tensor
    })

    # return an optional dict
    return output

```

--------------------------------

### Install Poetry Dependency Manager

Source: https://unbabel.github.io/OpenKiwi/installation.html

Install Poetry, the recommended build system and dependency manager for OpenKiwi development. Use this command to get Poetry via its official installation script.

```bash
curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python
```

--------------------------------

### Install OpenKiwi as a Library

Source: https://unbabel.github.io/OpenKiwi/installation.html

Use this command to install OpenKiwi for use as a Python library. This is the simplest method for integrating OpenKiwi into your projects.

```bash
pip install openkiwi
```

--------------------------------

### TLM System Checkpoint Loading Example

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/tlm_system/index.html

Example of how to restore custom data saved during checkpointing. Usually not required as Lightning handles training state restoration.

```python
def on_load_checkpoint(self, checkpoint):
    # 99% of the time you don't need to implement this method
    self.something_cool_i_want_to_save = checkpoint['something_cool_i_want_to_save']

```

--------------------------------

### Install OpenKiwi Package

Source: https://unbabel.github.io/OpenKiwi/index.html

Use this command to install the OpenKiwi package. This is the primary method for adding OpenKiwi to your project.

```bash
pip install openkiwi

```

--------------------------------

### Install OpenKiwi with Optuna Search (Poetry)

Source: https://unbabel.github.io/OpenKiwi/installation.html

Install OpenKiwi with optional Optuna hyperparameter search capabilities using Poetry. This command enables the search pipeline by specifying the 'search' extra.

```bash
poetry install -E search
```

--------------------------------

### Get Examples Per Field

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/data/corpus.html

Returns a dictionary mapping field names to their corresponding examples. Useful for accessing data organized by linguistic fields.

```python
def examples_per_field(self):
        examples = {
            field: examples
            for (field, _), examples in zip(
                self.dataset_fields, self.fields_examples
            )
        }
        return examples
```

--------------------------------

### Install MLFlow

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/loggers/index.html

Install the MLFlow library using pip. This is a prerequisite for using the MLFlowTrackingLogger.

```bash
pip install mlflow

```

--------------------------------

### Load Model and Predict Examples

Source: https://unbabel.github.io/OpenKiwi/source/kiwi.predictors.html

Demonstrates how to load a pre-trained model using kiwi.load_model and then use the predictor to generate predictions for a given set of examples. The examples are structured as a dictionary with source, target, and alignment fields.

```python
import kiwi
predictor = kiwi.load_model('tests/toy-data/models/nuqe.torch')
src = ['a b c', 'd e f g']
tgt = ['q w e r', 't y']
align = ['0-0 1-1 1-2', '1-1 3-0']
examples = {kiwi.constants.SOURCE: src,
                kiwi.constants.TARGET: tgt,
                kiwi.constants.ALIGNMENTS: align}
predictor.predict(examples)

```

--------------------------------

### Run OpenKiwi from Command Line (Local Install)

Source: https://unbabel.github.io/OpenKiwi/installation.html

After installing OpenKiwi as a local package, you can run the kiwi command from your terminal. This command should display a help message, confirming the installation.

```bash
python kiwi -h
```

```bash
kiwi -h
```

--------------------------------

### Install OpenKiwi with Optuna Search (Pip)

Source: https://unbabel.github.io/OpenKiwi/installation.html

Install OpenKiwi with optional Optuna hyperparameter search capabilities using pip. This command includes dependencies required for the search pipeline.

```bash
pip install openkiwi[search]
```

--------------------------------

### Install OpenKiwi Local Package Dependencies

Source: https://unbabel.github.io/OpenKiwi/installation.html

After installing Poetry, run this command within your virtual environment to install all project dependencies specified in pyproject.toml. This is for developing OpenKiwi locally.

```bash
poetry install
```

--------------------------------

### Run OpenKiwi from Command Line

Source: https://unbabel.github.io/OpenKiwi/index.html

Execute OpenKiwi directly from your command line after installation.

```bash
kiwi

```

--------------------------------

### Install OpenKiwi with MLflow Integration (Poetry)

Source: https://unbabel.github.io/OpenKiwi/installation.html

Install OpenKiwi with optional MLflow integration using Poetry. This command enables MLflow features by specifying the 'mlflow' extra.

```bash
poetry install -E mlflow
```

--------------------------------

### TLMSystem Training Step Example

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/tlm_system/index.html

Example implementation of the training_step method for the TLMSystem class. This method computes the training loss and logs metrics. It demonstrates how to perform a forward pass, calculate loss, and structure the return dictionary for logging and progress bar display.

```python
def training_step(self, batch, batch_idx):
    x, y, z = batch

    # implement your own
    out = self(x)
    loss = self.loss(out, x)

    logger_logs = {'training_loss': loss} # optional (MUST ALL BE TENSORS)

    # if using TestTubeLogger or TensorBoardLogger you can nest scalars
    logger_logs = {'losses': logger_logs} # optional (MUST ALL BE TENSORS)

    output = {
        'loss': loss, # required
        'progress_bar': {'training_loss': loss}, # optional (MUST ALL BE TENSORS)
        'log': logger_logs
    }

    # return a dict
    return output
```

--------------------------------

### Setup Output Directory

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/lib/utils.html

Creates or verifies an output directory, optionally constructing it from experiment and run IDs if not provided. Raises an error if the directory is required but missing.

```python
import argparse
from pathlib import Path

def setup_output_directory(
    output_dir,
    run_uuid=None,
    experiment_id=None,
    create=True
):
    """
    Sets up the output directory. This means either creating one, or
    verifying that the provided directory exists. Output directories
    are created using the run and experiment ids.

    Args:
        output_dir (str): The target output directory
        run_uuid : The current hash of the current run.
        experiment_id: The id of the current experiment
        create (bool): Boolean indicating whether to create a new folder.
    """
    if not output_dir:
        if experiment_id is None or run_uuid is None:
            raise argparse.ArgumentError(
                message='Please specify an output directory (--output-dir).',
                argument=output_dir,
            )
        output_path = Path('runs', str(experiment_id), str(run_uuid))
        output_dir = str(output_path)

    if create:
        Path(output_dir).mkdir(parents=True, exist_ok=True)
    elif not Path(output_dir).exists():
        raise FileNotFoundError(
            'Output directory does not exist: {}'.format(output_dir)
        )

    return output_dir
```

--------------------------------

### Initiate Search from File Programmatically

Source: https://unbabel.github.io/OpenKiwi/usage.html

Loads the search configuration from a file and starts the Optuna hyperparameter search. Requires importing the search function.

```python
from kiwi.lib.search import search_from_file

optuna_study = search_from_file('config/search.yaml')

```

--------------------------------

### Example Validation Step Implementation

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/tlm_system/index.html

A detailed example of a validation_step implementation, including loss calculation, logging images, and accuracy computation. Ensure all returned values are tensors.

```python
# CASE 1: A single validation dataset
def validation_step(self, batch, batch_idx):
    x, y = batch

    # implement your own
    out = self(x)
    loss = self.loss(out, y)

    # log 6 example images
    # or generated text... or whatever
    sample_imgs = x[:6]
    grid = torchvision.utils.make_grid(sample_imgs)
    self.logger.experiment.add_image('example_images', grid, 0)

    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)

    # all optional...
    # return whatever you need for the collation function validation_epoch_end
    output = OrderedDict({
        'val_loss': loss_val,
        'val_acc': torch.tensor(val_acc), # everything must be a tensor
    })

    # return an optional dict
    return output

```

--------------------------------

### setup

Source: https://unbabel.github.io/OpenKiwi/source/kiwi.lib.html

Analyzes pipeline options and configures the necessary requirements for running the prediction pipeline, such as setting up the output directory, random seeds, and the execution device.

```APIDOC
## setup

### Description
Analyze pipeline options and set up requirements to running the prediction pipeline. This includes setting up the output directory, random seeds and the device where predictions are run.

### Parameters
#### Path Parameters
- **options** (Namespace) - Required - Pipeline specific options

### Returns
- **output_dir** (str) - Path to output directory
```

--------------------------------

### Setup Training Pipeline Environment

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/lib/train.html

Configures the output directory, logging, random seeds, and device (GPU/CPU) for the training pipeline. Logs essential run information.

```python
output_dir = setup_output_directory(
        output_dir,
        tracking_logger.run_uuid,
        tracking_logger.experiment_id,
        create=True,
    )
    configure_logging(output_dir=output_dir, debug=debug, quiet=quiet)
    configure_seed(seed)

    logging.info("This is run ID: {}".format(tracking_logger.run_uuid))
    logging.info(
        "Inside experiment ID: {} ({})".format(
            tracking_logger.experiment_id, tracking_logger.experiment_name
        )
    )
    logging.info("Local output directory is: {}".format(output_dir))
    logging.info(
        "Logging execution to MLflow at: {}".format(
            tracking_logger.get_tracking_uri()
        )
    )

    if gpu_id is not None and gpu_id >= 0:
        torch.cuda.set_device(gpu_id)
        logging.info("Using GPU: {}".format(gpu_id))
    else:
        logging.info("Using CPU")

    logging.info(
        "Artifacts location: {}".format(
            tracking_logger.get_artifact_uri()
        )
    )

    return output_dir
```

--------------------------------

### Example Test DataLoader Implementation

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/qe_system/index.html

This example demonstrates how to implement a test DataLoader for a PyTorch model, including dataset loading, transformations, and batching. It's useful when a test dataset and test_step are required.

```python
def test_dataloader(self):
    transform = transforms.Compose([transforms.ToTensor(),
                                    transforms.Normalize((0.5,), (1.0,))])
    dataset = MNIST(root='/path/to/mnist/', train=False, transform=transform,
                    download=True)
    loader = torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=self.batch_size,
        shuffle=False
    )

    return loader
```

--------------------------------

### Install OpenKiwi with MLflow Integration (Pip)

Source: https://unbabel.github.io/OpenKiwi/installation.html

Install OpenKiwi with optional MLflow integration using pip. This command includes the necessary dependencies for leveraging MLflow capabilities.

```bash
pip install openkiwi[mlflow]
```

--------------------------------

### Predict with Examples

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/predictors/predictor.html

Generates predictions for a list of examples using the loaded QE model. It validates input, prepares data, and returns predictions categorized by level (e.g., word, sentence).

```python
def predict(self, examples, batch_size=1):
    """Create Predictions for a list of examples.

       Args:
         examples: A dict  mapping field names to the
           list of raw examples (strings).
         batch_size: Batch Size to use. Default 1.

       Returns:
         A dict mapping prediction levels
         (word, sentence ..) to the model predictions
         for each example.

       Raises:
         Exception: If an example has an empty string
           as `source` or `target` field.

       Example:
         >>> import kiwi
         >>> predictor = kiwi.load_model('tests/toy-data/models/nuqe.torch')
         >>> src = ['a b c', 'd e f g']
         >>> tgt = ['q w e r', 't y']
         >>> align = ['0-0 1-1 1-2', '1-1 3-0']
         >>> examples = {kiwi.constants.SOURCE: src,
                         kiwi.constants.TARGET: tgt,
                         kiwi.constants.ALIGNMENTS: align}
         >>> predictor.predict(examples)
         {'tags': [[0.4760947525501251,
            0.47569847106933594,
            0.4948718547821045,
            0.5305878520011902],
           [0.5105430483818054, 0.5252899527549744]]}
    """
    if not examples:
        return defaultdict(list)
    if self.fields is None:
        raise Exception('Missing fields object.')

    if not examples.get(const.SOURCE):
        raise KeyError('Missing required field "{}"'.format(const.SOURCE))
    if not examples.get(const.TARGET):
        raise KeyError('Missing required field "{}"'.format(const.TARGET))

    if not all(
        [s.strip() for s in examples[const.SOURCE] + examples[const.TARGET]]
    ):
        raise Exception(
            'Empty String in {} or {} field found!'.format(
                const.SOURCE, const.TARGET
            )
        )
    fields = [(name, self.fields[name]) for name in examples]

    field_examples = [
        Example.fromlist(values, fields)
        for values in zip(*examples.values())
    ]

    dataset = QEDataset(field_examples, fields=fields)

    return self.run(dataset, batch_size)

```

--------------------------------

### Test Step Implementation Example (Multiple Dataloaders)

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/qe_system/index.html

Illustrates the signature for a test_step when multiple test datasets are used, highlighting the use of dataset_idx to differentiate data sources.

```python
# CASE 2: multiple test datasets
def test_step(self, batch, batch_idx, dataset_idx):
    # dataset_idx tells you which dataset this is.


```

--------------------------------

### Download and Run SMT Models

Source: https://unbabel.github.io/OpenKiwi/_sources/reproduce.rst.txt

Use this command to download the SMT models and execute the run script. Ensure OpenKiwi is installed and the development data is available.

```bash
wget https://github.com/Unbabel/OpenKiwi/releases/download/0.1.1/en_de.smt_models.zip && unzip -n en_de.smt_models.zip && ./run_smt.sh
```

--------------------------------

### Training Step with Multiple Optimizers

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/qe_system/index.html

Example of how to handle multiple optimizers within the `training_step` by checking the `optimizer_idx` parameter.

```python
# Multiple optimizers (e.g.: GANs)
def training_step(self, batch, batch_idx, optimizer_idx):
    if optimizer_idx == 0:
        # do training_step with encoder
    if optimizer_idx == 1:
        # do training_step with decoder
```

--------------------------------

### Train Models using CLI

Source: https://unbabel.github.io/OpenKiwi/cli/train.html

Invoke the 'kiwi train' command with a model configuration file and optional parameters to start training.

```bash
kiwi train --config {model_config_file} [OPTS]

```

--------------------------------

### Setup Prediction Pipeline

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/lib/predict.html

Analyzes pipeline options to set up requirements for running predictions, including output directory, random seeds, and device configuration.

```python
def setup(options):
    """
    Analyze pipeline options and set up requirements to running
    the prediction pipeline. This includes setting up the output
    directory, random seeds and the device where predictions are run.

    Args:
        options(Namespace): Pipeline specific options

    """
    logger.debug("Setting up predict..")
    output_dir = setup_output_directory(options.output_dir)
    configure_seed(options.seed)
    configure_device(options.gpu_id)
    configure_logging(options.log_level)
    save_config_file(options, output_dir)
    return output_dir
```

--------------------------------

### Download and Run NMT Models

Source: https://unbabel.github.io/OpenKiwi/_sources/reproduce.rst.txt

Use this command to download the NMT models and execute the run script. Ensure OpenKiwi is installed and the development data is available.

```bash
wget https://github.com/Unbabel/OpenKiwi/releases/download/0.1.1/en_de.nmt_models.zip && unzip -n en_de.nmt_models.zip && ./run_nmt.sh
```

--------------------------------

### Evaluation Configuration File Example

Source: https://unbabel.github.io/OpenKiwi/configuration/evaluate.html

This YAML configuration specifies the paths to gold standard files and one or more directories containing predicted outputs for evaluation.

```yaml
gold_files:
    source_tags: data/WMT20/sentence_level/en_de.nmt/dev.source_tags
    target_tags: data/WMT20/sentence_level/en_de.nmt/dev.tags
    sentence_scores: data/WMT20/sentence_level/en_de.nmt/dev.hter

# Two configuration options:
# 1. (Recommended) Pass the root folders where the predictions live,
#    with the standard file names
predicted_dir:
    # The evaluation pipeline supports evaluating multiple predictions at the same time
    #   by passing the folders as a list
    - runs/0/4aa891368ff4402fa69a4b081ea2ba62
    - runs/0/e9200ada6dc84bfea807b3b02b9c7212

# 2. Configure each predicted file separately
# predicted_files:
#     source_tags:
#         - runs/0/4aa891368ff4402fa69a4b081ea2ba62/source_tags
#         - runs/0/e9200ada6dc84bfea807b3b02b9c7212/source_tags
#     # (Recommended) Pass the predicted `targetgaps_tags` file as `target_tags`;
#     #   the target and gap tags will be separated and evaluated separately as well as jointly
#     target_tags:
#         - runs/0/4aa891368ff4402fa69a4b081ea2ba62/targetgaps_tags
#         - runs/0/e9200ada6dc84bfea807b3b02b9c7212/targetgaps_tags
#     # Alternatively:
#     # target_tags:
#     #   - runs/0/4aa891368ff4402fa69a4b081ea2ba62/target_tags
#     #   - runs/0/e9200ada6dc84bfea807b3b02b9c7212/target_tags
#     # gap_tags:
#     #   - runs/0/4aa891368ff4402fa69a4b081ea2ba62/gap_tags
#     #   - runs/0/e9200ada6dc84bfea807b3b02b9c7212/gap_tags
#     sentence_scores:
#         - runs/0/4aa891368ff4402fa69a4b081ea2ba62/sentence_scores
#         - runs/0/e9200ada6dc84bfea807b3b02b9c7212/sentence_scores

```

--------------------------------

### Initialize Corpus

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/data/corpus.html

Creates a Corpus instance by providing lists of field values per example and dataset fields. Used for custom corpus creation.

```python
class Corpus:
    def __init__(self, fields_examples=None, dataset_fields=None):
        """Create a Corpus by specifying examples and fields.
        Arguments:
            fields_examples: A list of lists of field values per example.
            dataset_fields: A list of pairs (field name, field object).
        Both lists have the same size (number of fields).
        """
        self.fields_examples = (
            fields_examples if fields_examples is not None else []
        )
        self.dataset_fields = (
            dataset_fields if dataset_fields is not None else []
        )
        self.number_of_examples = (
            len(self.fields_examples[0]) if self.fields_examples else 0
        )
```

--------------------------------

### kiwi.lib.search.setup_run

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/lib/search/index.html

Sets up the output directory structure for Optuna search outputs.

```APIDOC
## kiwi.lib.search.setup_run

### Description
Set up the output directory structure for the Optuna search outputs.

### Parameters
#### Path Parameters
* **directory** (Path) - Description: 
* **seed** (int) - Description: 
* **debug** (bool) - Optional - Description: 
* **quiet** (bool) - Optional - Description: 

### Returns
Path
```

--------------------------------

### setup_run

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/lib/train/index.html

Prepares the environment and resources for executing the training pipeline.

```APIDOC
## setup_run

### Description
Prepare for running the training pipeline.

### Parameters
#### Request Body
- **config** (RunConfig) - Options for each run.
- **debug** (bool) - Optional. Enables debug mode.
- **quiet** (bool) - Optional. Suppresses output.
- **anchor_dir** (Path) - Optional. Directory to anchor run artifacts.

### Returns
- **Tuple[Path, Optional[MLFlowTrackingLogger]]** - A tuple containing the output directory path and an MLFlow tracking logger if configured.
```

--------------------------------

### setup_run

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/lib/search/index.html

Configures the output directory structure for Optuna search outputs.

```APIDOC
## setup_run

### Description
Sets up the output directory structure for the Optuna search outputs.

### Method
Not applicable (Python function)

### Parameters
#### Path Parameters
- **directory** (Path) - Required - The base directory for setting up the run.
- **seed** (int) - Required - The seed for reproducibility.
#### Query Parameters
- **debug** (bool) - Optional - Enables debug mode.
- **quiet** (bool) - Optional - Suppresses output.

### Response
- **Path** - The path to the set up output directory.
```

--------------------------------

### run

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/lib/train/index.html

Instantiates the system based on configuration and commences training.

```APIDOC
## run

### Description
Instantiate the system according to the configuration and train it.

### Parameters
#### Request Body
- **config** (Configuration) - The overall configuration for the training process.
- **system_type** (Union[Type[TLMSystem], Type[QESystem]]) - Optional. The type of system to instantiate for training (defaults to QESystem).

### Returns
- **TrainRunInfo** - Encapsulates relevant information about the training run.
```

--------------------------------

### kiwi.lib.train.setup

Source: https://unbabel.github.io/OpenKiwi/source/kiwi.lib.html

Analyzes pipeline options and sets up requirements for running the training pipeline. This includes setting up the output directory, random seeds, and the device(s) where training is run.

```APIDOC
## kiwi.lib.train.setup

### Description
Analyzes pipeline options and sets up requirements for running the training pipeline. This includes setting up the output directory, random seeds and the device(s) where training is run.

### Parameters
#### Path Parameters
- **output_dir** (string) - Required - Path to directory to use or None, in which case one is created automatically.
- **seed** (int) - Optional - Random seed for all random engines (Python, PyTorch, NumPy).
- **gpu_id** (int) - Optional - GPU number to use or None to use the CPU.
- **debug** (bool) - Optional - Whether to increase the verbosity of output messages.
- **quiet** (bool) - Optional - Whether to decrease the verbosity of output messages. Takes precedence over debug.

### Returns
- **output_dir** (string) - Path to output directory
```

--------------------------------

### setup_run

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/lib/predict/index.html

Prepare for running the prediction pipeline.

```APIDOC
## setup_run

### Description
Prepare for running the prediction pipeline.

### Method
`setup_run(config: RunConfig, quiet=False, debug=False, anchor_dir: Path = None) → Path`

### Parameters
#### Request Body
* **config** (RunConfig) - Description: Configuration for the run.
* **quiet** (bool) - Optional - Description: If true, suppress output.
* **debug** (bool) - Optional - Description: If true, enable debug logging.
* **anchor_dir** (Path) - Optional - Description: Directory to anchor the run.
```

--------------------------------

### setup_output_directory

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/lib/utils/index.html

Set up the output directory by creating it or verifying its existence, using run and experiment IDs.

```APIDOC
## setup_output_directory

### Description
Set up the output directory.
This means either creating one, or verifying that the provided directory exists. Output directories are created using the run and experiment ids.

### Parameters
* **output_dir** - The target output directory.
* **run_uuid** - Optional - The hash of the current run.
* **experiment_id** - Optional - The id of the current experiment.
* **create** (bool) - Optional - Whether to create the directory.

### Returns
* str - The path to the resolved output directory.
```

--------------------------------

### TrackingLogger Start Nested Run

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/loggers.html

Starts a nested run within the TrackingLogger. This is a convenience method that calls configure with nest_run=True.

```python
def start_nested_run(self, run_name=None):
        return self.configure(
            run_uuid=run_name, experiment_name=None, nest_run=True
        )
```

--------------------------------

### kiwi.lib.predict.setup_run

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/lib/predict/index.html

Prepares the prediction pipeline by setting up the output directory, random seeds, and loggers.

```APIDOC
## kiwi.lib.predict.setup_run

### Description
Prepares for running the prediction pipeline. This includes setting up the output directory, random seeds, and loggers.

### Parameters

#### Arguments
* **config** (RunConfig) - configuration options.
* **quiet** (bool) - Optional - whether to suppress info log messages.
* **debug** (bool) - Optional - whether to additionally log debug messages (:param:`quiet` has precedence)
* **anchor_dir** (Path) - Optional - directory to use as root for paths.

### Returns

* **Path** - the resolved path to the output directory.

### Related
* `kiwi.lib.evaluate`
* `kiwi.lib.pretrain`
```

--------------------------------

### Install Poetry using Pip (Not Recommended)

Source: https://unbabel.github.io/OpenKiwi/installation.html

An alternative method to install Poetry using pip. This is not recommended as it may conflict with local dependencies.

```bash
pip install poetry
```

--------------------------------

### QESystem Configuration Helper Methods

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/qe_system/index.html

Methods for setting configuration options and performing consistency checks.

```APIDOC
## Configuration Helper Methods

### `set_config_options(_optimizer_config: optimizers.OptimizerConfig = None, _batch_size: BatchSizeConfig = None, _num_data_workers: int = None, _data_config: WMTQEDataset.Config = None_)`

Allows setting specific configuration options dynamically.

### `map_name_to_class(_cls, _v_)`

Maps a class name to a class object.

### `check_consistency(_cls, _v_, _values_)`

Checks for consistency among configuration values.

### `check_model_requirement(_cls, _v_, _values_)`

Checks if model requirements are met.

### `check_batching(_cls, _v_)`

Checks batching configuration.
```

--------------------------------

### TLM System Checkpoint Saving Example

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/tlm_system/index.html

Example of how to save custom data within a PyTorch Lightning checkpoint. Typically not needed for standard training state.

```python
def on_save_checkpoint(self, checkpoint):
    # 99% of use cases you don't need to implement this method
    checkpoint['something_cool_i_want_to_save'] = my_cool_pickable_object

```

--------------------------------

### Train from Configuration File

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/lib/train.html

Loads training options from a specified configuration file and initiates the training process. Ensure the configuration file is correctly formatted.

```python
def train_from_file(filename):
    """
    Loads options from a config file and calls the training procedure.

    Args:
        filename (str): filename of the configuration file
    """
    parser = build_parser()
    options = parser.parse_config_file(filename)
    return train_from_options(options)
```

--------------------------------

### Run OpenKiwi from Command Line (Library Install)

Source: https://unbabel.github.io/OpenKiwi/installation.html

If installed as a library, you can run the kiwi command directly from your terminal. This is useful for accessing command-line utilities provided by OpenKiwi.

```bash
kiwi
```

--------------------------------

### Start Nested MLflow Run

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/loggers.html

Starts a new MLflow run that is nested within the current active run. Useful for organizing sub-tasks within a larger experiment.

```python
def start_nested_run(self, run_name=None):
    return mlflow.start_run(run_name=run_name, nested=True)
```

--------------------------------

### Initialize PipelineParser

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/cli/better_argparse.html

Sets up a parser for pipeline configurations, including options for I/O, general settings, logging, and model selection.

```python
self._parser = configargparse.get_argument_parser(
                self.name,
                add_help=False,
                prog='kiwi {}'.format(self.name),
                config_file_parser_class=configargparse.YAMLConfigFileParser,
                ignore_unknown_config_file_keys=True,
            )
            self._parsers[name] = self._parser
            self.add_config_option(self._parser)

            if add_io_options:
                opts.io_opts(self._parser)
            if add_general_options:
                opts.general_opts(self._parser)
            if add_logging_options:
                opts.logging_opts(self._parser)
            if add_save_load_options:
                opts.save_load_opts(self._parser)

            if options_fn is not None:
                options_fn(self._parser)

            if model_parsers is not None:
                group = self._parser.add_argument_group('models')
                group.add_argument(
                    '--model',
                    required=True,
                    choices=self._models.keys(),
                    help="Use 'kiwi {} --model <model> --help' for specific "
                    "model options.".format(self.name),
                )
```

--------------------------------

### Initiate Search from Configuration Dictionary Programmatically

Source: https://unbabel.github.io/OpenKiwi/usage.html

Loads a configuration from a file into a dictionary and then initiates the Optuna hyperparameter search. This method allows for manipulation of the configuration before search.

```python
from kiwi.lib.search import search_from_configuration
from kiwi.lib.utils import file_to_configuration

configuration_dict = file_to_configuration('config/search.yaml')
optuna_study = search_from_configuration(configuration_dict)

```

--------------------------------

### Custom Autograd Function Example

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/utils/tensors/index.html

Example of defining a custom autograd.Function in PyTorch by subclassing Function. This involves implementing the forward and backward passes to define custom operations and their gradients. Use the apply method to invoke the custom function.

```python
class Exp(Function):

    @staticmethod
    def forward(ctx, i):
        result = i.exp()
        ctx.save_for_backward(result)
        return result

    @staticmethod
    def backward(ctx, grad_output):
        result, = ctx.saved_tensors
        return grad_output * result

#Use it by calling the apply method:
output = Exp.apply(input)
```

--------------------------------

### Initialize NuQE Model

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/models/nuqe.html

Creates an instance of the NuQE model with specified vocabulary and configuration options. Ensure all required options are provided.

```python
    @staticmethod
    def from_options(vocabs, opts):
        model = NuQE(
            vocabs=vocabs,
            predict_target=opts.predict_target,
            predict_gaps=opts.predict_gaps,
            predict_source=opts.predict_source,
            source_embeddings_size=opts.source_embeddings_size,
            target_embeddings_size=opts.target_embeddings_size,
            hidden_sizes=opts.hidden_sizes,
            bad_weight=opts.bad_weight,
            window_size=opts.window_size,
            max_aligned=opts.max_aligned,
            dropout=opts.dropout,
            embeddings_dropout=opts.embeddings_dropout,
            freeze_embeddings=opts.freeze_embeddings,
        )
        return model
```

--------------------------------

### kiwi.data.utils.save_file

Source: https://unbabel.github.io/OpenKiwi/source/kiwi.data.html

Saves data to a file with specified token and example separators.

```APIDOC
## save_file(file_path, data, token_sep=' ', example_sep='\n')

### Description
Saves data to a file.

### Parameters
#### Path Parameters
- **file_path** (str or Path) - The path to the file to save.
- **data** (list) - The data to save.
- **token_sep** (str) - Separator for tokens (default: ' ').
- **example_sep** (str) - Separator for examples (default: '\n').
```

--------------------------------

### Setup Output Directory and Logging

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/lib/predict.html

Configures the output directory, logging, random seed, and device for the prediction pipeline. It logs the pipeline options and the local output directory. Optionally saves the configuration file.

```python
output_dir = setup_output_directory(
        options.output_dir,
        options.run_uuid,
        experiment_id=None,
        create=True,
    )
    configure_logging(
        output_dir=output_dir, debug=options.debug, quiet=options.quiet
    )
    configure_seed(options.seed)
    configure_device(options.gpu_id)

    logger.info(pformat(vars(options)))
    logger.info("Local output directory is: {}".format(output_dir))

    if options.save_config:
        save_config_file(options, options.save_config)

    del options.output_dir  # FIXME: remove this after making sure no other
    # place uses it! # noqa
    return output_dir
```

--------------------------------

### QEDataset.split

Source: https://unbabel.github.io/OpenKiwi/source/kiwi.data.html

Creates train-test(-valid?) splits from the instance’s examples.

```APIDOC
## QEDataset.split

### Description
Create train-test(-valid?) splits from the instance’s examples.

### Method
Instance Method

### Parameters
#### Path Parameters
* **split_ratio** (float or List of floats) - Optional - A number [0, 1] denoting the amount of data to be used for the training split (rest is used for validation), or a list of numbers denoting the relative sizes of train, test and valid splits respectively. If the relative size for valid is missing, only the train-test split is returned. Default is 0.7 (for the train set).
* **stratified** (bool) - Optional - Whether the sampling should be stratified. Default is False.
* **strata_field** (str) - Optional - Name of the examples Field stratified over. Default is ‘label’ for the conventional label field.
* **random_state** (tuple) - Optional - The random seed used for shuffling. A return value of random.getstate().

### Response
#### Success Response
- **Datasets** - Datasets for train, validation, and test splits in that order, if the splits are provided.

### Response Example
```
(Dataset, Dataset, Dataset)
```
```

--------------------------------

### Compose Configuration Files

Source: https://unbabel.github.io/OpenKiwi/configuration/index.html

Demonstrates how to compose configurations by referencing other YAML files. This allows for modular and dynamic configuration management.

```yaml
defaults:
 - data: wmt19.qe.en_de

```

--------------------------------

### Iterate through corpus examples

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/data/corpus.html

Provides an iterator for the corpus, yielding data.Example objects.

```python
def __iter__(self):
        for j in range(self.number_of_examples):
            fields_values_for_example = [
                self.fields_examples[i][j]
                for i in range(len(self.dataset_fields))
            ]
            yield data.Example.fromlist(
                fields_values_for_example, self.dataset_fields
            )
```

--------------------------------

### kiwi.lib.search.search_from_configuration

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/lib/search/index.html

Executes the entire training pipeline using provided configuration options.

```APIDOC
## kiwi.lib.search.search_from_configuration

### Description
Run the entire training pipeline using the configuration options received.

### Parameters
#### Request Body
* **configuration_dict** (dict) - Description: dictionary with options.

### Returns
object with training information.
```

--------------------------------

### Get Best Stats

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/trainers/callbacks.html

Returns the statistics associated with the best performing model iteration.

```python
def best_stats(self):
        stats, _ = self.best_stats_and_path()
        return stats
```

--------------------------------

### Run Training Experiments with Parameter Combinations

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/lib/search.html

Parses configuration, splits options, and iterates through all combinations of meta-options to run training experiments. It constructs full argument lists for each experiment run.

```python
def run(options, extra_options):
    config_parser = configargparse.YAMLConfigFileParser()
    config_options = config_parser.parse(Path(options.config).read_text())
    meta, fixed_options = split_options(config_options)

    # Run for each combination of arguments
    fixed_args = [options.model_name] + extra_options
    if options.experiment_name:
        fixed_args += parser.convert_item_to_command_line_arg(
            None, 'experiment-name', options.experiment_name
        )

    meta_keys = meta.keys()
    meta_values = meta.values()
    for values in itertools.product(*meta_values):
        assert len(meta_keys) == len(values)
        run_args = []
        for key, value in zip(meta_keys, values):
            action = get_action(key)
            run_args.extend(
                parser.convert_item_to_command_line_arg(action, key, str(value))
            )
        full_args = fixed_args + run_args + fixed_options
        train.main(full_args)
```

--------------------------------

### Get MLflow Tracking URI

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/loggers.html

Retrieves the current MLflow tracking server URI.

```python
@staticmethod
def get_tracking_uri():
    return mlflow.get_tracking_uri()
```

--------------------------------

### Get Metrics Ordering

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/models/predictor_estimator.html

Returns the ordering method for metrics. This specific implementation returns 'max'.

```python
return max
```

--------------------------------

### Get label ID from name

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/models/linear/label_dictionary.html

Retrieves the integer ID associated with a given label name.

```python
    def get_label_id(self, name):
        """Get label id from name."""
        return self[name]

```

--------------------------------

### Get label name from ID

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/models/linear/label_dictionary.html

Retrieves the label name associated with a given integer ID.

```python
    def get_label_name(self, label_id):
        """Get label name from id."""
        return self.names[label_id]

```

--------------------------------

### Evaluate Model Performance from Configuration

Source: https://unbabel.github.io/OpenKiwi/usage.html

Load a configuration and evaluate model predictions against a reference file. This method returns a detailed report.

```python
from kiwi.lib.evaluate import evaluate_from_configuration
from kiwi.lib.utils import file_to_configuration

configuration_dict = file_to_configuration('config/evaluate.yaml')
report = evaluate_from_configuration(configuration_dict)
print(report)

```

--------------------------------

### Reset PearsonMetric

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/metrics/metrics.html

Resets the lists storing predictions and target values. This is necessary before starting a new calculation.

```python
    def reset(self):
        self.predictions = []
        self.target = []
```

--------------------------------

### Get Worst Stats

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/trainers/callbacks.html

Returns the statistics of the worst performing model iteration currently stored in the history.

```python
def worst_stats(self):
        if self.best_stats_summary:
            return self.best_stats_summary[0][0]
        else:
            return None
```

--------------------------------

### Get MLflow Artifact URI

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/loggers.html

Retrieves the URI for accessing artifacts logged to the active MLflow run.

```python
@staticmethod
def get_artifact_uri():
    return mlflow.get_artifact_uri()
```

--------------------------------

### Get Active Experiment Name

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/loggers.html

Retrieves the name of the MLflow experiment associated with the current active run.

```python
@property
def experiment_name(self):
    # return MlflowClient().get_experiment(self.experiment_id).name
    return self._experiment_name
```

--------------------------------

### QESystem Initialization and Configuration

Source: https://unbabel.github.io/OpenKiwi/autoapi/kiwi/systems/qe_system/index.html

The QESystem class is initialized with a configuration object and an optional data configuration. It inherits from Serializable and LightningModule, providing a structured way to create ABCs.

```APIDOC
## QESystem

### Description
Helper class that provides a standard way to create an ABC using inheritance.

### Class Signature
`QESystem(_config, _data_config: WMTQEDataset.Config = None_)`

### Inheritance
Bases: `kiwi.systems._meta_module.Serializable`, `pytorch_lightning.LightningModule`
```

--------------------------------

### Configure Logging

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/lib/utils.html

Sets up the logging format, level (INFO, DEBUG, or WARNING), and optionally adds a file handler for log output.

```python
import logging
from time import gmtime
from pathlib import Path

def configure_logging(output_dir=None, debug=False, quiet=False):
    """
    Configure the logger. Sets up the log format, logging level
    and output directory of logging.

    Args:
        output_dir: The directory where log output will be stored.
            Defaults to None.
        debug (bool): Change logging level to debug.
        quiet (bool): Change logging level to warning to supress info logs.
    """
    logging.Formatter.converter = gmtime
    logging.Formatter.default_msec_format = '%s.%03d'
    log_format = '%(asctime)s [%(name)s %(funcName)s:%(lineno)s] %(message)s'
    if logging.getLogger().handlers:
        log_formatter = logging.Formatter(log_format)
        for handler in logging.getLogger().handlers:
            handler.setFormatter(log_formatter)
    else:
        logging.basicConfig(level=logging.INFO, format=log_format)

    log_level = logging.INFO
    if debug:
        log_level = logging.DEBUG
    if quiet:
        log_level = logging.WARNING

    logging.getLogger().setLevel(log_level)
    if output_dir is not None:
        fh = logging.FileHandler(str(Path(output_dir, 'output.log')))
        fh.setLevel(log_level)
        logging.getLogger().addHandler(fh)
```

--------------------------------

### Get Active Experiment ID

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/loggers.html

Retrieves the identifier of the MLflow experiment associated with the current active run.

```python
@property
def experiment_id(self):
    return mlflow.tracking.fluent.active_run().info.experiment_id
```

--------------------------------

### Get Active Run UUID

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/loggers.html

Retrieves the unique identifier (UUID) of the currently active MLflow run.

```python
@property
def run_uuid(self):
    return mlflow.tracking.fluent.active_run().info.run_uuid
```

--------------------------------

### Compute MovingSkipsAtQuality

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/metrics/metrics.html

Computes the fraction of skips and the average quality of skipped examples. Returns None for fraction if no skips occurred.

```python
def compute(self):
        if not self.skipped:
            return None, 0.0
        return (
            self.skipped / self.data_size,
            self.cumulative_qual / self.skipped,
        )
```

--------------------------------

### Load Trainer from Directory

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/trainers/trainer.html

Creates and returns a Trainer instance by loading its state from a specified directory. This includes loading the model, optimizer, and trainer's internal state.

```python
    @classmethod
    def from_directory(cls, directory, device_id=None):
        logger.info('Loading training state from {}'.format(directory))
        root_path = Path(directory)

        model_path = root_path / const.MODEL_FILE
        model = Model.create_from_file(model_path)

        if device_id is not None:
            model.to(device_id)

        optimizer_path = root_path / const.OPTIMIZER
        optimizer_dict = load_torch_file(str(optimizer_path))

        optimizer = optimizer_class(optimizer_dict['name'])(
            model.parameters(), lr=0.0
        )
        optimizer.load_state_dict(optimizer_dict['state_dict'])

        trainer = cls(model, optimizer, checkpointer=None)
        trainer_path = root_path / const.TRAINER
        state = load_torch_file(str(trainer_path))
        trainer.__dict__.update(state)
        return trainer
```

--------------------------------

### Get Best Iteration Path

Source: https://unbabel.github.io/OpenKiwi/_modules/kiwi/trainers/callbacks.html

Returns the file path of the best performing model iteration based on saved statistics.

```python
def best_iteration_path(self):
        _, path = self.best_stats_and_path()
        return path
```
Tokens:
Raw