Marian

https://marian-nmt.github.io/
Admin
Marian is an efficient Neural Machine Translation framework written in C++ that provides fast...
Tokens:1,807,046
Snippets:7,763
Trust Score:9.7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Start Training Using a Configuration File

Source: https://marian-nmt.github.io/examples/mtm2017/intro

Resume or start a training process using a previously generated configuration file. This is a convenient way to manage complex training setups.

```bash
./marian-dev/build/marian -c model/config.yml

```

--------------------------------

### Install Intel MKL for CPU Builds

Source: https://marian-nmt.github.io/docs

Commands to add the Intel repository and install MKL libraries on Ubuntu.

```bash
wget -qO- 'https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB' | sudo apt-key add -
sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
sudo apt-get update
sudo apt-get install intel-mkl-64bit-2020.0-088
```

--------------------------------

### Initialize Model Directory and Start Basic Training

Source: https://marian-nmt.github.io/examples/mtm2017/intro

Create an output directory for the model and start a basic training process. Inspect the --help option for default model configurations like batch size and encoder type.

```bash
mkdir -p model

./marian-dev/build/marian \
  --train-set data/corpus.clean.bpe.ro data/corpus.clean.bpe.en \
  --model model/model.npz \
  --disp-freq 100

```

--------------------------------

### Clone Marian examples repository

Source: https://marian-nmt.github.io/examples/mtm2018-labs

Download the Marian examples repository to access training scripts and data. This is the first step in setting up a training environment.

```bash
git clone https://github.com/marian-nmt/marian-examples
cd marian-examples/tools
make
cd ../training-basics
```

--------------------------------

### Install Ubuntu dependencies

Source: https://marian-nmt.github.io/quickstart

Install essential packages for compiling Marian on Ubuntu.

```bash
sudo apt-get install git cmake build-essential
```

--------------------------------

### Example Commands for Training Factored Models

Source: https://marian-nmt.github.io/docs/api/factors

Provides example MARIAN NMT commands for training factored models in various configurations.

```APIDOC
## Examples

### Example 1: Factors on both source and target, sum combination, no tied embeddings, no lemma dependency
```bash
path_to/build/marian -t corpus.fact.{src,trg} \
                     -v vocab.{src,trg}.fsv
```

### Example 2: Factors only on source, concat combination, tied embeddings
```bash
path_to/build/marian -t corpus.fact.src corpus.trg \
                     -v vocab.src.fsv vocab.trg.yml \
                     --factors-combine concat \
                     --factors-dim-emb 8 \
                     --tied-embeddings-all
```

### Example 3: Factors only on target, sum combination, tied embeddings, soft-transformer-layer lemma dependency
```bash
path_to/build/marian -t corpus.src corpus.fact.trg \
                     -v vocab.src.yml vocab.fsv.trg \
                     --tied-embeddings \
                     --lemma-dependency soft-transformer-layer
```
```

--------------------------------

### Training Start Message

Source: https://marian-nmt.github.io/docs/api/_sources/api/program_listing_file_src_training_scheduler.h.rst.txt

Logs an informational message indicating that training has started.

```cpp
void started() { LOG(info, "Training started"); }
```

--------------------------------

### Display Marian help

Source: https://marian-nmt.github.io/examples/mtm2017/intro

Verify the installation by displaying the available command-line options.

```bash
./marian --help
```

--------------------------------

### Get Guided Alignment

Source: https://marian-nmt.github.io/docs/api/_sources/api/program_listing_file_src_data_corpus_base.h.rst.txt

Retrieves the guided alignment information for the batch. The alignment is stored in a flattened format.

```cpp
const std::vector<WordAlignment>& getGuidedAlignment() const { return guidedAlignment_; }  // [dimSrcWords, dimBatch, dimTrgWords] flattened
```

--------------------------------

### Download tutorial scripts

Source: https://marian-nmt.github.io/examples/mtm2017/intro

Clone the necessary repositories for preprocessing and subword splitting.

```bash
cd ../..

git clone https://github.com/marian-nmt/mtm2017-tutorial
git clone https://github.com/marian-nmt/moses-scripts
git clone https://github.com/rsennrich/subword-nmt
```

--------------------------------

### Get Alignments from Attention Layer

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_models_s2s.h.html

Helper function to retrieve alignments from a specific attention layer within the stacked RNN cells. Used for guided alignment purposes.

```cpp
// helper function for guided alignment
  virtual const std::vector<Expr> getAlignments(int i = 0) override {
    auto att
        = rnn_->at(0)->as<rnn::StackedCell>()->at(i + 1)->as<rnn::Attention>();
    return att->getAlignments();
  }
```

--------------------------------

### Navigate to working directory

Source: https://marian-nmt.github.io/examples/mtm2017/intro

Change the current working directory to the tutorial folder.

```bash
cd mtm2017-marian
```

--------------------------------

### Get Number of TSV Input Fields

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_data_corpus_base.cpp.html

Calculates the number of input fields in a TSV file, excluding fields used for guided alignment and data weighting if they are enabled.

```cpp
size_t CorpusBase::getNumberOfTSVInputFields(Ptr<Options> options) {
  if(options->get<bool>("tsv", false)) {
    size_t n = options->get<size_t>("tsv-fields", 0);
    if(n > 0 && options->get("guided-alignment", std::string("none")) != "none")
      --n;
    if(n > 0 && options->hasAndNotEmpty("data-weighting"))
      --n;
    return n;
  }
  return 0;
}
```

--------------------------------

### Initialize and Parse Options with ConfigParser

Source: https://marian-nmt.github.io/docs/api/api/classmarian_1_1ConfigParser.html

Demonstrates how to instantiate ConfigParser and parse command-line arguments for a server application.

```cpp
int main(int argc, char* argv[]) { ConfigParser cp(cli::mode::translation); cp.addOption<int>(”–port”, // option name “Server Options”, // option group name “Port for server.”, // help string 5678); // default value auto opts = cp.parseOptions(argc,argv,true); // ‘true’ for validation … 
```

--------------------------------

### Get Alignments Helper Function

Source: https://marian-nmt.github.io/docs/api/_sources/api/program_listing_file_src_models_transformer.h.rst.txt

Retrieves alignments, typically used for guided alignment. The return type `const vector<>` is noted as potentially incorrect and may need adjustment to `const&` for efficiency.

```cpp
    // helper function for guided alignment
    // @TODO: const vector<> seems wrong. Either make it non-const or a const& (more efficient but dangerous)
    virtual const std::vector<Expr> getAlignments(int /*i*/ = 0) override {
      return alignments_; // [tgt index][beam depth, max src length, batch size, 1]
    }
  
```

--------------------------------

### Get Device IDs for Training

Source: https://marian-nmt.github.io/docs/api/_sources/api/program_listing_file_src_common_config.cpp.rst.txt

Parses device specifications from options to determine which devices to use for training. Supports CPU threads and GPU devices, with complex logic for multi-node and multi-GPU setups.

```cpp
std::vector<DeviceId> Config::getDevices(Ptr<Options> options,
                                            size_t myMPIRank /*= 0*/,
                                            size_t numMPINodes /*= 1*/ ) {
  std::vector<DeviceId> deviceIds;
  if(options->devices.size() > 0) {
    // --devices used
    if(numMPINodes > 1 && options->devices.size() % numMPINodes == 0) {
      // multi-node, devices shared across nodes
      size_t devicesPerNode = options->devices.size() / numMPINodes;
      for(size_t i = 0; i < devicesPerNode; ++i)
        deviceIds.push_back(options->devices[myMPIRank * devicesPerNode + i]);
    }
    else if(numMPINodes > 1 && options->devices.size() == options->numDevices * numMPINodes) {
      // multi-node, devices split across nodes
      for(size_t i = 0; i < options->numDevices; ++i)
        deviceIds.push_back(options->devices[myMPIRank * options->numDevices + i]);
    }
    else {
      // single node or devices not split
      for(size_t i = 0; i < options->devices.size(); ++i)
        deviceIds.push_back(options->devices[i]);
    }
  }
  else if(options->numDevices > 0) {
    // --num-devices used
    for(size_t i = 0; i < options->numDevices; ++i)
      deviceIds.push_back(i);
  }
  else {
    // default: use device 0
    deviceIds.push_back(0);
  }
  return deviceIds;
}
```

--------------------------------

### Initialize Working Directory

Source: https://marian-nmt.github.io/examples/exploration

Commands to create and enter the project directory.

```bash
mkdir ape-explore
cd ape-explore
```

--------------------------------

### Tests and Examples Structure

Source: https://marian-nmt.github.io/docs/api/structure.html

The 'examples' and 'tests' subdirectories within 'marian/src' contain basic unit tests and examples demonstrating Marian's functionality with common datasets like Iris and MNIST. These examples showcase various model types and training configurations.

```plaintext
marian/src
├── examples
└── tests
```

--------------------------------

### Start Marian Training Using Configuration File

Source: https://marian-nmt.github.io/examples/mtm2017/complex

Initiate the training process for a Marian NMT model by referencing a previously generated configuration file. This simplifies the command line arguments.

```bash
./marian-dev/build/marian -c model.deep/config.yml
```

--------------------------------

### Struct Example Functions

Source: https://marian-nmt.github.io/docs/api/api/structmarian_1_1data_1_1Example.html

Documentation for the public functions of the 'Example' struct.

```APIDOC
## Struct Example Functions

### Public Functions
- `size_t getId() const`:
  Returns the ID of the example. This is a const member function.
```

--------------------------------

### Set Up Sgd Optimizer and Update Parameters

Source: https://marian-nmt.github.io/docs/api/_sources/graph.md.txt

Configure the Sgd optimizer with a learning rate and update the model parameters. This example also shows how to retrieve the updated value of a parameter.

```cpp
// set up Sgd optimiser with 0.005 learning rate
auto opt = Optimizer<Sgd>(0.005);
// update parameters
opt->update(graph);
// get the new value of x
std::vector<float> v;
x->val()->get(v);
std::cout<<"x="<<v[0]<<std::endl;
// The output is: x=1.98708
```

--------------------------------

### Compile Protobuf from Source

Source: https://marian-nmt.github.io/docs

Build and install Protobuf from source code for custom environment configurations.

```bash
wget https://github.com/protocolbuffers/protobuf/releases/download/v2.6.1/protobuf-cpp-2.6.1.zip
unzip protobuf-cpp-2.6.1.zip
cd protobuf-2.6.1
./autogen.sh
./configure --prefix $(pwd)
make -j4
make install
```

--------------------------------

### Initialize Graphs and Options

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_training_graph_group.cpp.html

Sets up the expression graphs, default element types, checkpointing, NaN checking, device assignment, and workspace size for each device. It also initializes optimizer shards and criterion functions.

```cpp
void GraphGroup::initGraphsAndOpts() {
  for(auto device : devices_) {
    auto graph = New<ExpressionGraph>();

    auto precisions = options_->get<std::vector<std::string>>("precision");
    Type parameterType = typeFromString(precisions[0]);

    graph->setDefaultElementType(parameterType);
    graph->setCheckpointing(options_->get<bool>("gradient-checkpointing"));

    if(options_->get<bool>("check-nan")) // @TODO: add to other places
      graph->setThrowNaN(true);

    graph->setDevice(device);

    graph->reserveWorkspaceMB(options_->get<size_t>("workspace"));

    graphs_.push_back(graph);

    optimizerShards_.push_back(Optimizer(options_));
    models_.push_back(models::createCriterionFunctionFromOptions(options_, models::usage::training));
  }
}

```

--------------------------------

### Struct Example Definition

Source: https://marian-nmt.github.io/docs/api/api/structmarian_1_1data_1_1Example.html

Documentation for the 'Example' struct, which inherits from std::vector<Data>.

```APIDOC
## Struct Example

### Description
Represents an example structure that inherits from `std::vector<Data>`.

### Inheritance
- **Base Type**: `public std::vector<Data>`

### Constructors
- `Example(std::vector<Data> &&_data_, size_t _id_)`
- `Example()`

### Public Members
- **id** (`size_t`): An identifier for the example.
```

--------------------------------

### Doxygen @warning Command Example

Source: https://marian-nmt.github.io/docs/api/_sources/doc_guide.rst.txt

Example of using the @warning command to add a warning message.

```text
@warning

```

--------------------------------

### Start Transformer Training

Source: https://marian-nmt.github.io/examples/mtm2019

Initiate the training process for a transformer-base model using a configuration file.

```bash
~/marian-dev/build/marian --task transformer-base -c config.yml
```

--------------------------------

### Doxygen @param Command Example

Source: https://marian-nmt.github.io/docs/api/_sources/doc_guide.rst.txt

Example of using the @param command to describe a function parameter.

```text
@param device a
            pointer to the
            device
```

--------------------------------

### Set Guided Alignment

Source: https://marian-nmt.github.io/docs/api/_sources/api/program_listing_file_src_data_corpus_base.h.rst.txt

Sets the guided alignment for the batch using move semantics for efficiency.

```cpp
void setGuidedAlignment(std::vector<WordAlignment>&& aln) override {
      guidedAlignment_ = std::move(aln);
    }
```

--------------------------------

### Checkout Tutorial Branch

Source: https://marian-nmt.github.io/examples/mtm2017/code

Commands to fetch and build the tutorial-nov-17 branch containing the pre-implemented sutskever model type.

```bash
cd marian-dev
cd build
git fetch origin tutorial-nov-17
git checkout tutorial-nov-17
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j
```

--------------------------------

### Link Error Example

Source: https://marian-nmt.github.io/docs/api/_sources/operators.md.txt

Example of a typical link error caused by a missing explicit template instantiation.

```text
.../src/tensors/tensor_operators.h:41: undefined reference to `void marian::gpu::Element<marian::functional::Assign< ... > ( ... )'
```

--------------------------------

### Doxygen @note Command Example

Source: https://marian-nmt.github.io/docs/api/_sources/doc_guide.rst.txt

Example of using the @note command to add a highlighted note message.

```text
@note this is named
      after an equivalent
      function in PyTorch
```

--------------------------------

### Class DummyInit

Source: https://marian-nmt.github.io/docs/api/_sources/api/classmarian_1_1inits_1_1DummyInit.rst.txt

Documentation for the DummyInit class, including inheritance and source file location.

```APIDOC
## Class DummyInit

### Description
DummyInit is a class defined in the Marian NMT framework, specifically within the graph node initializers module.

### Inheritance
- **Base Type**: marian::inits::NodeInitializer

### Source File
- src/graph/node_initializers.cpp
```

--------------------------------

### Define Example and Data Structures in C++

Source: https://marian-nmt.github.io/docs/api/_sources/api/program_listing_file_src_examples_mnist_dataset.h.rst.txt

Defines custom types for data representation, including `Data` (vector of floats), `Labels` (vector of indices), and `Example` (a vector of `Data` with an ID). It also includes iterator types for examples.

```cpp
#pragma once

#include <algorithm>
#include <cassert>
#include <fstream>
#include <iostream>
#include <memory>
#include <string>
#include <vector>

#include "common/definitions.h"
#include "common/shape.h"
#include "data/batch.h"
#include "data/dataset.h"
#include "data/vocab.h"

namespace marian {
namespace data {

typedef std::vector<float> Data;
typedef std::vector<IndexType> Labels;
struct Example : public std::vector<Data> { // a std::vector<Data> with a getId() method
  size_t id;
  size_t getId() const { return id; }
  Example(std::vector<Data>&& data, size_t id) : std::vector<Data>(std::move(data)), id(id) {}
  Example() : id(SIZE_MAX) {}
};
typedef std::vector<Example> Examples;

typedef Examples::const_iterator ExampleIterator;

class Input {
private:
  Shape shape_;
  Ptr<Data> data_;

public:
  typedef Data::iterator iterator;
  typedef Data::const_iterator const_iterator;

  Input(const Shape& shape) : shape_(shape), data_(new Data(shape_.elements(), 0.0f)) {}

  Data::iterator begin() { return data_->begin(); }
  Data::iterator end() { return data_->end(); }

  Data::const_iterator begin() const { return data_->cbegin(); }
  Data::const_iterator end() const { return data_->cend(); }

  Data& data() { return *data_; }

  Shape shape() const { return shape_; }

  size_t size() const { return data_->size(); }
};

class DataBatch : public Batch {
private:
  std::vector<Input> inputs_;

public:

  std::vector<Input>& inputs() { return inputs_; }

  const std::vector<Input>& inputs() const { return inputs_; }

  void push_back(Input input) { inputs_.push_back(input); }

  virtual std::vector<Ptr<Batch>> split(size_t /*n*/, size_t /*sizeLimit*/) override { ABORT("Not implemented"); }

  Data& features() { return inputs_[0].data(); }

  Data& labels() { return inputs_.back().data(); }

  size_t size() const override { return inputs_.front().shape()[0]; }

  void setGuidedAlignment(std::vector<WordAlignment>&&) override {
    ABORT("Guided alignment in DataBatch is not implemented");
  }
  void setDataWeights(const std::vector<float>&) override {
    ABORT("Data weighting in DataBatch is not implemented");
  }
};

class Dataset : public DatasetBase<Example, ExampleIterator, DataBatch>, public RNGEngine {
protected:
  Examples examples_;

public:
  Dataset(const std::vector<std::string>& paths, Ptr<Options> options)
      : DatasetBase(paths, options) {}

  virtual void loadData() = 0;

  iterator begin() override { return ExampleIterator(examples_.begin()); }

  iterator end() override { return ExampleIterator(examples_.end()); }

  void shuffle() override { std::shuffle(examples_.begin(), examples_.end(), eng_); }

  batch_ptr toBatch(const Examples& batchVector) override {
    int batchSize = (int)batchVector.size();

    std::vector<int> maxDims;
    for(auto& ex : batchVector) {
      if(maxDims.size() < ex.size())
        maxDims.resize(ex.size(), 0);
      for(size_t i = 0; i < ex.size(); ++i) {
        if(ex[i].size() > (size_t)maxDims[i])
          maxDims[i] = (int)ex[i].size();
      }
    }

    batch_ptr batch(new DataBatch());
    std::vector<Input::iterator> iterators;
    for(auto& m : maxDims) {
      batch->push_back(Shape({batchSize, m}));
      iterators.push_back(batch->inputs().back().begin());
    }

    for(auto& ex : batchVector) {
      for(size_t i = 0; i < ex.size(); ++i) {
        Data d = ex[i];
        d.resize(maxDims[i], 0.0f);
        iterators[i] = std::copy(d.begin(), d.end(), iterators[i]);
      }
    }
    return batch;
  }
};

class MNISTData : public Dataset {
private:
  const int IMAGE_MAGIC_NUMBER;
  const int LABEL_MAGIC_NUMBER;

public:
  MNISTData(std::vector<std::string> paths,
              std::vector<Ptr<Vocab>> /*vocabs*/ = {},
              Ptr<Options> options = nullptr)
      : Dataset(paths, options), IMAGE_MAGIC_NUMBER(2051), LABEL_MAGIC_NUMBER(2049) {
    loadData();
  }

  virtual ~MNISTData(){}

  void loadData() override {
    ABORT_IF(paths_.size() != 2, "Paths to MNIST data files are not specified");

    auto features = ReadImages(paths_[0]);
    auto labels = ReadLabels(paths_[1]);

```

--------------------------------

### Display Directory Structure

Source: https://marian-nmt.github.io/examples/mtm2019

Shows the file layout of the downloaded tutorial directory.

```text
marian-tutorial-mtm19
├ 1_decoding
│   ├ data
│   │   ├ newstest2013.de
│   │   ├ newstest2013.en
│   │   ├ newstest2014.de
│   │   └ newstest2014.en
│   ├ model.npz
│   ├ run-me.sh
│   └ vocab.spm
├ 2_training
│   ├ config.yml
│   ├ download-data.sh
│   ├ run-me.sh
│   └ train.log
├ 3_student
│   ├ data
│   │   ├ newstest2014.bpe.en
│   │   ├ newstest2014.de
│   ├ lex.s2t
│   ├ model.student.base
│   │   └ model.npz
│   ├ model.student.small
│   │   └ model.npz
│   ├ model.student.small.aan.nogate.noffn
│   │   └ model.npz
│   ├ run-me.sh
│   └ vocab.ende.yml
├ detruecase.perl
└ multi-bleu.perl
```

--------------------------------

### Configure and Start Advanced Model Training

Source: https://marian-nmt.github.io/examples/mtm2017/intro

Initiate advanced training with a configuration file, specifying model type, validation sets, vocabulary paths, layer normalization, dimensions, batch fitting, workspace, dropout rates, exponential smoothing, early stopping, and logging options.

```bash
./marian-dev/build/marian \
  --type s2s \
  --train-set data/corpus.clean.bpe.ro data/corpus.clean.bpe.en \
  --valid-set data/newsdev2016.bpe.ro data/newsdev2016.bpe.en \
  --vocabs data/vocab.ro.yml data/vocab.en.yml \
  --model model/model.npz \
  --layer-normalization \
  --dim-vocabs 66000 50000 \
  --mini-batch-fit --workspace 3000 \
  --dropout-rnn 0.2 --dropout-src 0.1 --exponential-smoothing \
  --early-stopping 5 --disp-freq 1000 \
  --log model/train.log --valid-log model/valid.log \
  --dump-config > model/config.yml

```

--------------------------------

### Doxygen @copydetails Command Example

Source: https://marian-nmt.github.io/docs/api/_sources/doc_guide.rst.txt

Example of using the @copydetails command to copy the detailed documentation from another object.

```text
@copydetails dot
```

--------------------------------

### LambdaInit Class

Source: https://marian-nmt.github.io/docs/api/api/classmarian_1_1inits_1_1LambdaInit.html

Documentation for the LambdaInit class, including its constructor and apply method.

```APIDOC
## Class LambdaInit

Defined in File node_initializers.cpp

### Inheritance Relationships
#### Base Type
* public marian::inits::NodeInitializer (Class NodeInitializer)

### Class Documentation
_class_`LambdaInit` : _public_ marian::inits::NodeInitializer

    
Public Functions 

`LambdaInit`(std::function<void(Tensor)> &&_lambda_)
     

void `apply`(Tensor _tensor_)
    
```

--------------------------------

### Doxygen @copybrief Command Example

Source: https://marian-nmt.github.io/docs/api/_sources/doc_guide.rst.txt

Example of using the @copybrief command to copy the brief description from another object.

```text
@copybrief slice
```

--------------------------------

### Download Tutorial Data

Source: https://marian-nmt.github.io/examples/mtm2019

Downloads and extracts the required models and datasets for the tutorial.

```bash
wget http://data.statmt.org/romang/marian-examples/marian-tutorial-mtm19.tgz
tar zxvf marian-tutorial-mtm19.tgz
cd marian-tutorial-mtm19
```

--------------------------------

### Doxygen @see Command Example

Source: https://marian-nmt.github.io/docs/api/_sources/doc_guide.rst.txt

Example of using the @see command to create a cross-reference to another documented item.

```text
@see reshape()
```

--------------------------------

### Doxygen @return Command Example

Source: https://marian-nmt.github.io/docs/api/_sources/doc_guide.rst.txt

Example of using the @return command to describe a function's return value.

```text
@return a pointer
           to the constant
           node
```

--------------------------------

### Install Protobuf Dependencies on Ubuntu

Source: https://marian-nmt.github.io/docs

Install required Protobuf packages based on the specific Ubuntu distribution version.

```bash
# Ubuntu 20.04 (Focal Fossa):
sudo apt-get install libprotobuf17 protobuf-compiler libprotobuf-dev

# Ubuntu 18.04 (Bionic Beaver):
sudo apt-get install libprotobuf10 protobuf-compiler libprotobuf-dev

# Ubuntu 16.04 LTS (Xenial Xerus):
sudo apt-get install libprotobuf9v5 protobuf-compiler libprotobuf-dev

# Ubuntu 14.04 LTS (Trusty Tahr):
sudo apt-get install libprotobuf8 protobuf-compiler libprotobuf-dev
```

--------------------------------

### Class DummyInit

Source: https://marian-nmt.github.io/docs/api/api/classmarian_1_1inits_1_1DummyInit.html

Documentation for the DummyInit class, which is defined in node_initializers.cpp and inherits from marian::inits::NodeInitializer.

```APIDOC
## Class DummyInit

### Description
DummyInit is a class used for node initialization in the Marian NMT framework. It inherits from the base class `marian::inits::NodeInitializer`.

### Inheritance
- **Base Type**: `marian::inits::NodeInitializer`

### Public Functions
- **apply(Tensor tensor)**: Applies the initialization logic to the provided tensor.
```

--------------------------------

### Load Configuration Files

Source: https://marian-nmt.github.io/docs/cmd/marian-scorer

Use -c or --config to specify one or more configuration files. Later files override earlier ones.

```bash
-c,--config VECTOR ...                Configuration file(s). If multiple, later overrides earlier
```

--------------------------------

### Initialize Loggers from Configuration

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_common_logging.cpp.html

Sets up general and validation loggers based on provided configuration. Handles log file paths, quiet mode, log level, and time zone settings.

```cpp
void createLoggers(const marian::Config* config) {
  std::vector<std::string> generalLogs;
  std::vector<std::string> validLogs;

  if(config && !config->get<std::string>("log").empty()) {
    generalLogs.push_back(config->get<std::string>("log"));
#ifndef _WIN32
    // can't open the same file twice in Windows for some reason
    validLogs.push_back(config->get<std::string>("log"));
#endif
  }

  // valid-log is available only for training
  if(config && config->has("valid-log") && !config->get<std::string>("valid-log").empty()) {
    validLogs.push_back(config->get<std::string>("valid-log"));
  }

  bool quiet = config && config->get<bool>("quiet");
  Logger general{createStderrLogger("general", "[%Y-%m-%d %T] %v", generalLogs, quiet)};
  Logger valid{createStderrLogger("valid", "[%Y-%m-%d %T] [valid] %v", validLogs, quiet)};

  if(config && config->has("log-level")) {
    std::string loglevel = config->get<std::string>("log-level");
    if(!setLoggingLevel(*general, loglevel))
      return;
    setLoggingLevel(*valid, loglevel);
  }

  if(config && !config->get<std::string>("log-time-zone").empty()) {
    std::string timezone = config->get<std::string>("log-time-zone");
#ifdef _WIN32
#define setenv(var, val, over) SetEnvironmentVariableA(var, val) // ignoring over flag
#endif
    setenv("TZ", timezone.c_str(), true);
    tzset();
  }

  setErrorHandlers();
}
```

--------------------------------

### Get Piece ID (Interop)

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_microsoft_sentencepiece.cpp.html

Interoperability function to get the ID of a SentencePiece token. Handles potential exceptions.

```cpp
int SentencePieceInteropGetPieceID(intptr_t object, char* word) {
  try {

```

--------------------------------

### Install Marian Toolkit

Source: https://marian-nmt.github.io/examples/exploration

Commands to clone, build, and compile the Marian NMT toolkit.

```bash
git clone https://github.com/marian-nmt/marian
cd marian
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j16
cd ../..
```

--------------------------------

### Doxygen @ref Command Example

Source: https://marian-nmt.github.io/docs/api/_sources/doc_guide.rst.txt

Example of using the @ref command to create a reference to another documented item by its identifier.

```text
@ref IndexType
```

--------------------------------

### Complete .fsv file example

Source: https://marian-nmt.github.io/docs/api/_sources/factors.md.txt

A full example of a .fsv vocabulary file including comments and both factors and lemmas sections.

```text
 # factors

_lemma

_c
c0 : _c
c1 : _c
c2 : _c
_has_c

_s
s0 : _s
s1 : _s
_has_s

 # lemmas

</s> : _lemma
<unk> : _lemma
, : _lemma _has_c _has_s
. : _lemma _has_c _has_s
the : _lemma _has_c _has_s
for : _lemma _has_c _has_s
```

--------------------------------

### Initialize Configuration from Parser

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_common_config.cpp.html

Initializes the configuration by cloning settings from a ConfigParser and setting up logging. It also logs the Marian version, command line, and hostname/PID.

```cpp
void Config::initialize(ConfigParser const& cp) {
  config_ = YAML::Clone(cp.getConfig());
  cli::mode mode = cp.getMode();

  createLoggers(this);

  // echo version and command line
  LOG(info, "[marian] Marian {}", buildVersion());
  std::string cmdLine = cp.cmdLine();
  std::string hostname;
  int pid;
  std::tie
  (hostname, pid) = utils::hostnameAndProcessId();
  LOG(info, "[marian] Running on {} as process {} with command line:", hostname, pid);
  LOG(info, "[marian] {}", cmdLine);

  // set random seed
  if(get<size_t>("seed") == 0) {
    seed = (size_t)time(0);
  } else {
    seed = get<size_t>("seed");
  }

  // load model parameters
  bool loaded = false;
  if(mode == cli::mode::translation || mode == cli::mode::server) {
    auto model = get<std::vector<std::string>>("models")[0];
    try {
      if(!get<bool>("ignore-model-config"))
        loaded = loadModelParameters(model);
    } catch(std::runtime_error& ) {
      LOG(info, "[config] No model configuration found in model file");
    }
  }
  // if cli::mode::training or cli::mode::scoring
  else {
    auto model = get<std::string>("model");
    if(filesystem::exists(model) && !get<bool>("no-reload")) {
      try {
        if(!get<bool>("ignore-model-config"))
          loaded = loadModelParameters(model);
      } catch(std::runtime_error&){
        LOG(info, "[config] No model configuration found in model file");
      }
    }
  }

  // guess --tsv-fields, i.e. the number of fields in a TSV input, if not set
  if(get<bool>("tsv") && get<size_t>("tsv-fields") == 0) {
    size_t tsvFields = 0;

    // use the length of --input-types if given
    auto inputTypes = get<std::vector<std::string>>("input-types");
    if(!inputTypes.empty()) {
      tsvFields = inputTypes.size();
    } else {
      if(loaded) {
        // model.npz has properly set vocab dimensions in special:model.yml,
        // so we may use them to determine the number of streams
        for(auto dim : get<std::vector<size_t>>("dim-vocabs"))
          if(dim != 0)  // language models have a fake extra vocab
            ++tsvFields;
        // For translation there is no target stream
        if((mode == cli::mode::translation || mode == cli::mode::server) && tsvFields > 1)
          --tsvFields;
      } else {
        // If parameters from model.npz special:model.yml were not loaded,
        // guess the number of inputs and outputs based on the model type name.
        // TODO: This is very britle, find a better solution
        auto modelType = get<std::string>("type");

        tsvFields = 1;
        if(modelType.find("multi-", 0) != std::string::npos)  // is a dual-source model
          tsvFields += 1;
        if(mode == cli::mode::training || mode == cli::mode::scoring)
          if(modelType.rfind("lm", 0) != 0)  // unless it is a language model
            tsvFields += 1;
      }

      // count fields with guided-alignment or data-weighting too
      if(mode == cli::mode::training) {
        if(has("guided-alignment") && get<std::string>("guided-alignment") != "none")
          tsvFields += 1;
        if(has("data-weighting") && !get<std::string>("data-weighting").empty())
          tsvFields += 1;
      }
    }

    config_["tsv-fields"] = tsvFields;
  }

  // ensures factors backward compatibility whilst keeping the more user friendly CLI
  if(get<std::string>("lemma-dependency").empty()) {
    YAML::Node config;
    int lemmaDimEmb = get<int>("lemma-dim-emb");
    if(lemmaDimEmb > 0) {
      config_["lemma-dependency"] = "re-embedding";
    } else if(lemmaDimEmb == -1) {
      config_["lemma-dependency"] = "lemma-dependent-bias";
    } else if(lemmaDimEmb == -2) {
      config_["lemma-dependency"] = "soft-transformer-layer";
    } else if(lemmaDimEmb == -3) {
      config_["lemma-dependency"] = "hard-transformer-layer";
    }
  }

  // echo full configuration
  log();

  // Log version of Marian that has been used to create the model.
  //
  // Key "version" is present only if loaded from model parameters and is not
  // related to --version flag
  if(has("version")) {

```

--------------------------------

### Function main()

Source: https://marian-nmt.github.io/docs/api/_sources/api/function_iris_8cpp_1ae66f6b31b5ad750f1fe042a706a4e3d4.rst.txt

Documentation for the main entry point function defined in iris.cpp.

```APIDOC
## main()

### Description
The main entry point function for the Iris example application.

### Defined in
src/examples/iris/iris.cpp
```

--------------------------------

### Function main(int, char **)

Source: https://marian-nmt.github.io/docs/api/api/function_marian__scorer_8cpp_1a3c04138a5bfe5d72780bb7e82a18e627.html

Documentation for the main entry point function found in marian_scorer.cpp.

```APIDOC
## Function main(int, char **)

### Description
This is the main entry point function defined in the file marian_scorer.cpp.

### File
marian_scorer.cpp

### Note
The documentation for this function is currently unavailable as the Doxygen XML output for the project could not be parsed.
```

--------------------------------

### Add Guided Alignments to Batch

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_data_corpus_base.cpp.html

Adds guided alignments to a batch if the batch vector has not been altered. This is crucial for maintaining alignment validity.

```cpp
void CorpusBase::addGuidedAlignmentsToBatch(Ptr<CorpusBatch> batch,
                                   const std::vector<Sample>& batchVector) {
  int dimBatch = (int)batch->size();
  std::vector<std::vector<size_t>> aligns;
  aligns.reserve(dimBatch);

  for(int b = 0; b < dimBatch; ++b) {
    // If the batch vector is altered within marian by, for example, case augmentation,
    // the guided alignments we received for this tuple cease to be valid.
    // Hence skip setting alignments for that sentence tuple..
    if (!batchVector[b].isAltered()) {
      aligns.push_back(std::move(batchVector[b].getAlignment()));
    }
  }
  batch->setGuidedAlignment(std::move(aligns));
}
```

--------------------------------

### Set Guided Alignment for Batch

Source: https://marian-nmt.github.io/docs/api/_sources/api/program_listing_file_src_data_corpus_base.h.rst.txt

Sets the guided alignment for a batch. This is typically used when processing batches that require specific alignment information.

```cpp
batch->setGuidedAlignment(std::move(alignment));
```

--------------------------------

### Function main(int, char **)

Source: https://marian-nmt.github.io/docs/api/api/function_marian__train_8cpp_1a3c04138a5bfe5d72780bb7e82a18e627.html

Documentation for the main entry point function defined in marian_train.cpp.

```APIDOC
## Function main(int, char **)

### Description
The main entry point for the Marian NMT training process, defined in marian_train.cpp.

### Parameters
- **argc** (int) - Number of command line arguments
- **argv** (char **) - Array of command line argument strings
```

--------------------------------

### Define Tuple Get and Top-K Functions

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_graph_expression_operators.h.html

Provides a templated function to get elements from a 2-element tuple and a function to perform top-k selection.

```cpp
typedef std::tuple<Expr, Expr> Expr2;

template <int I>
Expr get(Expr2 tuple) { return std::get<I>(tuple); }

Expr2 topk(Expr a, int k, int axis, bool descending = true);

```

--------------------------------

### Define L_SURROGATE_START

Source: https://marian-nmt.github.io/docs/api/_sources/api/define_unicode__conversions_8h_1a70e12779192e9c461d0b145b23b03a73.rst.txt

Documentation for the L_SURROGATE_START macro defined in the unicode_conversions.h header file.

```APIDOC
## Define L_SURROGATE_START

### Description
Represents the starting value for Unicode surrogate pairs as defined in the Microsoft Unicode Conversions library.

### File
src/microsoft/unicode_conversions.h
```

--------------------------------

### Main Training Entry Point in marian_train.cpp

Source: https://marian-nmt.github.io/docs/api/api/program_listing_file_src_command_marian_train.cpp.html

Initializes training options and selects between synchronous or asynchronous graph groups based on configuration.

```cpp
#include <signal.h>
#include "marian.h"

#include "common/signal_handling.h"
#include "training/graph_group_async.h"
#include "training/graph_group_singleton.h"
#include "training/graph_group_sync.h"
#include "training/training.h"

#include "3rd_party/ExceptionWithCallStack.h"

int main(int argc, char** argv) {
  using namespace marian;
  auto options = parseOptions(argc, argv, cli::mode::training);

  // --sync-sgd always selects SyncGraphGroup
  //
  // If given, then this implementation is used for all combinations of (single, multiple) MPI
  // processes x (single, multiple) GPUs per MPI process.  This variant is presently up-to-date and
  // best supported.
  if(options->get<bool>("sync-sgd")) { // @TODO: make default
    LOG(info, "Using synchronous SGD");
    New<Train<SyncGraphGroup>>(options)->run();
  }
  else {
    auto devices = Config::getDevices(options);
    if(devices.size() == 1) {
      LOG(info, "[training] Using single-device training");
      New<Train<SyncGraphGroup>>(options)->run();
      // New<Train<SingletonGraph>>(options)->run(); // kept for reference
    } else {
      LOG(info, "Using asynchronous training");
      New<Train<AsyncGraphGroup>>(options)->run();
    }
  }
  // If we exit due to a graceful exit request via SIGTERM, exit with 128 + SIGTERM,
  // as suggested for bash in http://tldp.org/LDP/abs/html/exitcodes.html. This allows parent
  // scripts to determine if training terminated naturally or via SIGTERM.
  // An alternative would be to exit with code 124, which is what the timeout command
  // returns for timeout -s SIGTERM <seconds> ...., because exiting after SIGTERM
  // is not technically a fatal error (which is what the 128+x convention usually
  // stands for).
  exit(getSignalFlag(SIGTERM) ? 128 + SIGTERM : EXIT_SUCCESS);
}
```
Tokens:
Raw