CTranslate2

https://opennmt.net/CTranslate2/
Admin
CTranslate2 is a fast inference engine for Transformer models supporting efficient execution,...
Tokens:86,077
Snippets:618
Trust Score:9.4
Update:1 month ago
Show doc for...
Tokens:
Raw
### Compile C++ example with custom install path

Source: https://opennmt.net/CTranslate2/quickstart.html

Configure CMake to use a custom CTranslate2 installation by setting CMAKE_PREFIX_PATH.

```bash
cmake -DCMAKE_PREFIX_PATH=$CTRANSLATE_INSTALL_PATH .
```

--------------------------------

### Compile and run C++ example

Source: https://opennmt.net/CTranslate2/quickstart.html

Build and run the C++ example using CMake and make.

```bash
cmake .
make
./main
```

--------------------------------

### Install Python packages

Source: https://opennmt.net/CTranslate2/quickstart.html

Install CTranslate2, OpenNMT-py version 2.x, and sentencepiece using pip.

```bash
pip install ctranslate2 OpenNMT-py==2.* sentencepiece
```

--------------------------------

### Install transformers and convert a model with ct2-transformers-converter

Source: https://opennmt.net/CTranslate2/guides/transformers.html

Install the Transformers library with PyTorch support, then convert a pretrained Transformers model (e.g., facebook/m2m100_418M) to a CTranslate2 model directory using the ct2-transformers-converter command.

```bash
pip install transformers[torch]
ct2-transformers-converter --model facebook/m2m100_418M --output_dir ct2_model

```

--------------------------------

### Install OpenNMT-tf and convert model with ct2-opennmt-tf-converter

Source: https://opennmt.net/CTranslate2/guides/opennmt_tf.html

Install OpenNMT-tf and run the converter with a YAML configuration file. The output directory is specified with --output_dir. Use ct2-opennmt-tf-converter -h to see other conversion options.

```bash
pip install OpenNMT-tf
ct2-opennmt-tf-converter --config config.yml --output_dir ct2_model

```

--------------------------------

### Compile and install CTranslate2

Source: https://opennmt.net/CTranslate2/quickstart.html

Build and install CTranslate2 from source using CMake. The library must be installed into a directory on CMAKE_PREFIX_PATH.

```bash
mkdir build && cd build
cmake ..
make -j4 install
```

--------------------------------

### Compile Python wrapper with pybind11

Source: https://opennmt.net/CTranslate2/installation.html

Compiles and installs the Python wrapper after the C++ library is installed. Requires Python development libraries. If the C++ library is in a custom directory, set CTRANSLATE2_ROOT when running setup.py and add the library path to LD_LIBRARY_PATH when running the application.

```bash
cd python
pip install -r install_requirements.txt
python setup.py bdist_wheel
pip install dist/*.whl

```

--------------------------------

### Compile C++ library with CMake

Source: https://opennmt.net/CTranslate2/installation.html

Compiles and installs the C++ library. Requires a compiler supporting C++17 and CMake 3.15 or greater. By default, the Intel MKL backend is used and must be installed separately.

```bash
mkdir build && cd build
cmake ..
make -j4
sudo make install
sudo ldconfig

```

--------------------------------

### Install CTranslate2 to custom directory

Source: https://opennmt.net/CTranslate2/quickstart.html

Install CTranslate2 to a custom directory by setting CTRANSLATE_INSTALL_PATH and passing CMAKE_INSTALL_PREFIX to CMake.

```bash
export CTRANSLATE_INSTALL_PATH=$(pwd)/install
cmake .. -DCMAKE_INSTALL_PREFIX=$CTRANSLATE_INSTALL_PATH
make -j4 install
```

--------------------------------

### as_integer_ratio example

Source: https://opennmt.net/CTranslate2/python/ctranslate2.specs.Activation.html

Shows the output of calling as_integer_ratio() on integer literals. The method returns a pair of integers representing the ratio in lowest terms with a positive denominator.

```python
>>> (10).as_integer_ratio()
(10, 1)
>>> (-10).as_integer_ratio()
(-10, 1)
>>> (0).as_integer_ratio()
(0, 1)
```

--------------------------------

### bit_length example

Source: https://opennmt.net/CTranslate2/python/ctranslate2.specs.Activation.html

Shows the output of bin() and bit_length() on the integer 37. bit_length() returns the number of bits necessary to represent the integer in binary.

```python
>>> bin(37)
'0b100101'
>>> (37).bit_length()
6
```

--------------------------------

### bit_count example

Source: https://opennmt.net/CTranslate2/python/ctranslate2.specs.Activation.html

Shows the output of bin() and bit_count() on the integer 13. bit_count() returns the number of ones in the binary representation of the absolute value.

```python
>>> bin(13)
'0b1101'
>>> (13).bit_count()
3
```

--------------------------------

### Instantiate and translate with ctranslate2.Translator

Source: https://opennmt.net/CTranslate2/python/ctranslate2.Translator.html

Shows how to create a Translator with a model directory and CPU device, then translate a batch of tokenized text. The example uses the translate_batch method with a list of token sequences.

```python
>>> translator = ctranslate2.Translator("model/", device="cpu")
>>> translator.translate_batch([["▁Hello", "▁world", "!"]])
```

--------------------------------

### Download and convert MBART-50 model

Source: https://opennmt.net/CTranslate2/guides/fairseq.html

Download and convert the MBART-50 multilingual translation model. The output directory is used in the subsequent translation example.

```shell
wget https://dl.fbaipublicfiles.com/fairseq/models/mbart50/mbart50.ft.nn.tar.gz
tar xf mbart50.ft.nn.tar.gz

ct2-fairseq-converter --data_dir mbart50.ft.nn/ --model_path mbart50.ft.nn/model.pt \
    --output_dir mbart50_ct2

```

--------------------------------

### Force translation start with target_prefix

Source: https://opennmt.net/CTranslate2/decoding.html

Forces the start of the translation using the target_prefix argument. The prefix changes the target context and the rest of the translation, as shown in the example output.

```python
results = translator.translate_batch(
    [tokenize(input)],
    target_prefix=[tokenize("Dieses Projekt ist auf das")],
)

print(detokenize(results[0].hypotheses[0]))

```

--------------------------------

### OpenNMTTFConverter.__init__()

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.html

Initializes an OpenNMTTFConverter instance.

```APIDOC
## OpenNMTTFConverter.__init__()

### Description
Initializes an OpenNMTTFConverter instance.

### Method
__init__

### Signature
`__init__(self, model_path: str, ...)`

### Parameters
- **model_path** (str) - Required - Path to the OpenNMT-tf model file.

### Returns
None

```

--------------------------------

### Generate text with transformer_lm

Source: https://opennmt.net/CTranslate2/guides/opennmt_py.html

Generate text with a converted OpenNMT-py transformer_lm model. Always include the start token <s> in the start tokens, as shown with the example batch containing "<s>" and "▁Hello".

```python
generator = ctranslate2.Generator(model_path)
generator.generate_batch([["<s>", "▁Hello"]])

```

--------------------------------

### Wav2Vec2Bert.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Wav2Vec2Bert.html

Initializes a Wav2Vec2Bert model from a converted model.

```APIDOC
## Wav2Vec2Bert.__init__

### Description
Initializes a Wav2Vec2Bert model from a converted model.

### Parameters
- **model_path** (str) - Required - Path to the CTranslate2 model directory.
- **device** (str) - Optional - Device to use (possible values are: cpu, cuda, auto). Default: 'cpu'.
- **device_index** (Union[int, List[int]]) - Optional - Device IDs where to place this model on. Default: 0.
- **compute_type** (Union[str, Dict[str, str]]) - Optional - Model computation type or a dictionary mapping a device name to the computation type (possible values are: default, auto, int8, int8_float32, int8_float16, int8_bfloat16, int16, float16, bfloat16, float32). Default: 'default'.
- **inter_threads** (int) - Optional - Number of workers to allow executing multiple batches in parallel. Default: 1.
- **intra_threads** (int) - Optional - Number of OpenMP threads per worker (0 to use a default value). Default: 0.
- **max_queued_batches** (int) - Optional - Maximum numbers of batches in the worker queue (-1 for unlimited, 0 for an automatic value). When the queue is full, future requests will block until a free slot is available. Default: 0.
- **flash_attention** (bool) - Optional - Run model with flash attention 2 for self-attention layer. Default: False.
- **tensor_parallel** (bool) - Optional - Run model with tensor parallel mode. Default: False.
- **files** (object) - Optional - Load model files from the memory. This argument is a dictionary mapping file names to file contents as file-like or bytes objects. If this is set, `model_path` acts as an identifier for this model. Default: None.

### Returns
None
```

--------------------------------

### OpenNMTPyConverter.__init__()

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.html

Initializes an OpenNMTPyConverter instance.

```APIDOC
## OpenNMTPyConverter.__init__()

### Description
Initializes an OpenNMTPyConverter instance.

### Method
__init__

### Signature
`__init__(self, model_path: str, ...)`

### Parameters
- **model_path** (str) - Required - Path to the OpenNMT-py model file.

### Returns
None

```

--------------------------------

### Wav2Vec2Spec.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.specs.Wav2Vec2Spec.html

Initializes a Wav2Vec2Spec language model specification with the provided parameters.

```APIDOC
## Wav2Vec2Spec.__init__

### Description
Initializes a language model specification with the given parameters.

### Method
Constructor

### Parameters
- **feat_layers** (int) - Required - Number of feature layers.
- **num_layers** (int) - Required - Number of layers.
- **num_heads** (int) - Required - Number of attention heads.
- **vocab_size** (int) - Required - Vocabulary size.
- **return_hidden** (bool) - Required - Whether to return hidden states.

### Returns
None

```

--------------------------------

### Install CTranslate2 with pip

Source: https://opennmt.net/CTranslate2/installation.html

Installs the CTranslate2 Python package from PyPI. Requires Python >= 3.9 and pip >= 19.3 for manylinux2014 wheels. GPU support is available on Linux and Windows with CUDA 12.x.

```bash
pip install ctranslate2
```

--------------------------------

### FairseqConverter.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.FairseqConverter.html

Initializes the Fairseq converter with the model path, data directory, and optional language and dictionary settings.

```APIDOC
## FairseqConverter.__init__

### Description
Initializes the Fairseq converter with the model path, data directory, and optional language and dictionary settings.

### Method
Constructor

### Signature
`__init__(model_path: str, data_dir: str, source_lang: Optional[str] = None, target_lang: Optional[str] = None, fixed_dictionary: Optional[str] = None, no_default_special_tokens: bool = False, user_dir: Optional[str] = None, unsafe_deserialization: bool = False)`

### Parameters
- **model_path** (str) - Required - Path to the Fairseq PyTorch model (.pt file).
- **data_dir** (str) - Required - Path to the Fairseq data directory containing vocabulary files.
- **source_lang** (Optional[str]) - Optional - Source language (may be required if not declared in the model).
- **target_lang** (Optional[str]) - Optional - Target language (may be required if not declared in the model).
- **fixed_dictionary** (Optional[str]) - Optional - Path to the fixed dictionary for multilingual models.
- **no_default_special_tokens** (bool) - Optional - Require all special tokens to be provided by the user (e.g. encoder end token, decoder start token).
- **user_dir** (Optional[str]) - Optional - Path to the user directory containing custom extensions.
- **unsafe_deserialization** (bool) - Optional - Allow unsafe pickle deserialization when loading trusted legacy checkpoints.
```

--------------------------------

### OpenAIGPT2Converter.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.OpenAIGPT2Converter.html

Initializes the OpenAI GPT-2 converter with the path to the model directory.

```APIDOC
## OpenAIGPT2Converter.__init__

### Description
Initializes the OpenAI GPT-2 converter with the path to the model directory.

### Method
Constructor

### Parameters
- **model_dir** (str) - Required - Path to the OpenAI GPT-2 model directory.

### Returns
None

```

--------------------------------

### Wav2Vec2.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Wav2Vec2.html

Initializes a Wav2Vec2 model from a converted model.

```APIDOC
## Wav2Vec2.__init__

### Description
Initializes a Wav2Vec2 model from a converted model.

### Parameters
- **model_path** (str) - Required - Path to the CTranslate2 model directory.
- **device** (str) - Optional - Device to use (possible values are: cpu, cuda, auto). Default: 'cpu'.
- **device_index** (Union[int, List[int]]) - Optional - Device IDs where to place this model on. Default: 0.
- **compute_type** (Union[str, Dict[str, str]]) - Optional - Model computation type or a dictionary mapping a device name to the computation type (possible values are: default, auto, int8, int8_float32, int8_float16, int8_bfloat16, int16, float16, bfloat16, float32). Default: 'default'.
- **inter_threads** (int) - Optional - Number of workers to allow executing multiple batches in parallel. Default: 1.
- **intra_threads** (int) - Optional - Number of OpenMP threads per worker (0 to use a default value). Default: 0.
- **max_queued_batches** (int) - Optional - Maximum numbers of batches in the worker queue (-1 for unlimited, 0 for an automatic value). When the queue is full, future requests will block until a free slot is available. Default: 0.
- **flash_attention** (bool) - Optional - run model with flash attention 2 for self-attention layer. Default: False.
- **tensor_parallel** (bool) - Optional - run model with tensor parallel mode. Default: False.
- **files** (object) - Optional - Load model files from the memory. This argument is a dictionary mapping file names to file contents as file-like or bytes objects. If this is set, `model_path` acts as an identifier for this model.

### Returns
None
```

--------------------------------

### num_languages (property)

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

Gets the number of languages supported.

```APIDOC
## num_languages (property)

### Description
Returns the number of languages supported.
```

--------------------------------

### device (property)

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

Gets the device this model is running on.

```APIDOC
## device (property)

### Description
Device this model is running on.
```

--------------------------------

### Wav2Vec2BertSpec.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.specs.Wav2Vec2BertSpec.html

Initializes a language model specification with the given parameters.

```APIDOC
## Wav2Vec2BertSpec.__init__

### Description
Initializes a language model specification.

### Method
Constructor

### Signature
__init__(_num_hidden_layers_, _num_adapter_layers_, _vocab_size_, _return_hidden_)

### Parameters
- **num_hidden_layers** (int) - Required - Number of hidden layers.
- **num_adapter_layers** (int) - Required - Number of adapter layers.
- **vocab_size** (int) - Required - Vocabulary size.
- **return_hidden** (bool) - Required - Whether to return hidden states.

```

--------------------------------

### Prompt caching with static_prompt for StableLM

Source: https://opennmt.net/CTranslate2/generation.html

Demonstrates using the static_prompt argument to cache the model state for a fixed system prompt, as with StableLM. The cache is unlimited and cleared only when the model is unloaded; on multiple GPUs each replica manages its own cache.

```python
import ctranslate2
import transformers

generator = ctranslate2.Generator("stablelm-ct2/", device="cpu")
tokenizer = transformers.AutoTokenizer.from_pretrained("stabilityai/stablelm-tuned-alpha-7b")

system_prompt = """<|SYSTEM|># StableLM Tuned (Alpha version)
- StableLM is a helpful and harmless open-source AI language model developed by StabilityAI.
- StableLM is excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
- StableLM is more than just an information source, StableLM is also able to write poetry, short stories, and make jokes.
- StableLM will refuse to participate in anything that could harm a human.
"""
system_prompt_tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(system_prompt))

prompt = "<|USER|>What's your mood today?<|ASSISTANT|>"
prompt_tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))

step_results = generator.generate_tokens(
    prompt=prompt_tokens,
    static_prompt=system_prompt_tokens,
    max_length=512,
    sampling_topk=10,
    sampling_temperature=0.7,
    end_token=[50278, 50279, 50277, 1, 0],
)
```

--------------------------------

### Encoder Class

Source: https://opennmt.net/CTranslate2/python/ctranslate2.html

Class for forward encoding of a batch of examples.

```APIDOC
## Encoder

### Description
Computes forward representations of a batch.

### Methods
- `Encoder.__init__()`
- `Encoder.forward_batch()`
- `Encoder.load_model()`
- `Encoder.unload_model()`

### Parameters
- `compute_type` (str) - The compute type.
- `device` (Device) - The device type.
- `device_index` (int) - The device index.
- `model_is_loaded` (bool) - Whether a model is loaded.
- `num_active_batches` (int) - Number of active batches.
- `num_encoders` (int) - Number of parallel encoders.
- `num_queued_batches` (int) - Number of queued batches.
- `tensor_parallel` (int) - Tensor parallelism size.
```

--------------------------------

### num_queued_batches (property)

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

Gets the number of batches waiting to be processed.

```APIDOC
## num_queued_batches (property)

### Description
Number of batches waiting to be processed.
```

--------------------------------

### n_mels (property)

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

Gets the dimension of mel input features.

```APIDOC
## n_mels (property)

### Description
Returns dimension of mel input features.
```

--------------------------------

### compute_type (property)

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

Gets the computation type used by the model.

```APIDOC
## compute_type (property)

### Description
Computation type used by the model.
```

--------------------------------

### Run CTranslate2 Docker image with --help

Source: https://opennmt.net/CTranslate2/installation.html

Runs the CTranslate2 Docker image to display the translator executable's help. The image entrypoint is the translator executable. Requires the NVIDIA Container Toolkit for GPU support.

```bash
docker run --rm ghcr.io/opennmt/ctranslate2:latest-ubuntu22.04-cuda11.2 --help
```

--------------------------------

### num_workers (property)

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

Gets the number of model workers backing this instance.

```APIDOC
## num_workers (property)

### Description
Number of model workers backing this instance.
```

--------------------------------

### OpusMTConverter.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.OpusMTConverter.html

Initializes the OPUS-MT converter with the path to the model directory.

```APIDOC
## OpusMTConverter.__init__

### Description
Initializes the OPUS-MT converter.

### Method
Constructor

### Signature
`__init__(model_dir: str)`

### Parameters
- **model_dir** (str) - Required - Path to the OPUS-MT model directory.

### Returns
None
```

--------------------------------

### device_index (property)

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

Gets the list of device IDs where this model is running on.

```APIDOC
## device_index (property)

### Description
List of device IDs where this model is running on.
```

--------------------------------

### num_active_batches (property)

Source: https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

Gets the number of batches waiting to be processed or currently processed.

```APIDOC
## num_active_batches (property)

### Description
Number of batches waiting to be processed or currently processed.
```

--------------------------------

### OpenMPI hostfile configuration

Source: https://opennmt.net/CTranslate2/parallel.html

Defines the hostfile for OpenMPI, listing IP addresses or DNS names with the number of GPUs (slots) per host.

```text
[ipaddress or dns] slots=nbGPU1
[other ipaddress or dns] slots=NbGPU2

```

--------------------------------

### Generator.__call__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.Generator.html

Generates sequences from a batch of start tokens. This method is not compatible with beam search.

```APIDOC
## Generator.__call__

### Description
Generates sequences from a batch of start tokens. This method is not compatible with beam search.

### Method
__call__

### Parameters
- **prompt** (List[List[str]]) - Required - Batch of start tokens. If the decoder starts from a special start token like <s>, this token should be added to this input.
- **max_batch_size** (int) - Optional - The maximum batch size.
- **batch_type** (str) - Optional - Whether `max_batch_size` is the number of "examples" or "tokens".
- **max_length** (int) - Optional - Maximum generation length.
- **min_length** (int) - Optional - Minimum generation length.
- **sampling_topk** (int) - Optional - Randomly sample predictions from the top K candidates.
- **sampling_topp** (float) - Optional - Keep the most probable tokens whose cumulative probability exceeds this value.
- **sampling_temperature** (float) - Optional - Sampling temperature to generate more random samples.
- **return_log_prob** (bool) - Optional - Include the token log probability in the result.
- **repetition_penalty** (float) - Optional - Penalty applied to the score of previously generated tokens (set > 1 to penalize).
- **no_repeat_ngram_size** (int) - Optional - Prevent repetitions of ngrams with this size (set 0 to disable).
- **disable_unk** (bool) - Optional - Disable the generation of the unknown token.
- **suppress_sequences** (List[List[int]]) - Optional - Disable the generation of some sequences of tokens.
- **end_token** (int) - Optional - Stop the decoding on one these tokens (defaults to the model EOS token).
- **static_prompt** (List[int]) - Optional - If the model expects a static prompt (a.k.a. system prompt) it can be set here to simplify the inputs and optionally cache the model state for this prompt to accelerate future generations.
- **cache_static_prompt** (bool) - Optional - Cache the model state after the static prompt and reuse it for future generations using the same static prompt.
- **callback** (Callable) - Optional - Optional function that is called for each generated token when obj:beam_size is 1. If the callback function returns `True`, the decoding will stop for this batch index.

### Returns
A generator iterator over `ctranslate2.GenerationStepResult` instances.

### Note
This generation method is not compatible with beam search which requires a complete decoding.
```

--------------------------------

### Translator.score_iterable

Source: https://opennmt.net/CTranslate2/python/ctranslate2.Translator.html

Scores an iterable of tokenized examples. Built on top of score_batch for efficient stream processing.

```APIDOC
## Translator.score_iterable

### Description
Scores an iterable of tokenized examples. This method is built on top of `ctranslate2.Translator.score_batch()` to efficiently score an arbitrarily large stream of data. It enables the following optimizations: stream processing (the iterable is not fully materialized in memory), parallel scoring (if the translator has multiple workers), asynchronous batch prefetching, and local sorting by length.

### Method
score_iterable

### Parameters
- **source** (Iterable[List[str]]) - Required - Iterable of source token sequences.
- **target** (Iterable[List[str]]) - Required - Iterable of target token sequences.
- **max_batch_size** (int) - Optional - The maximum batch size.
- **batch_type** (str) - Optional - Whether `max_batch_size` is the number of "examples" or "tokens".
- **kwargs** - Optional - Additional keyword arguments passed to `score_batch`.

### Returns
An iterable of scoring results.
```

--------------------------------

### Run application with mpirun for tensor parallelism

Source: https://opennmt.net/CTranslate2/parallel.html

Launches the application in multiprocess mode using mpirun to enable tensor parallelism across GPUs.

```bash
mpirun -np nbGPUExpected -hostfile hostfile python3 script

```

--------------------------------

### generate_batch

Source: https://opennmt.net/CTranslate2/python/ctranslate2.Generator.html

Generates sequences from a batch of start tokens using the CTranslate2 generator. This method is not compatible with beam search.

```APIDOC
## Generator

### Description
Generates sequences from a batch of start tokens using the CTranslate2 generator. This method is not compatible with beam search.

### Method
async generate_batch

### Parameters
- **start_tokens** (List[List[str]]) - Required - Batch of start tokens. If the decoder starts from a special start token like <s>, this token should be added to this input.
- **max_batch_size** (int) - Optional - The maximum batch size.
- **batch_type** (str) - Optional - Whether `max_batch_size` is the number of "examples" or "tokens".
- **asynchronous** (bool) - Optional - Whether to return an async generator.
- **beam_size** (int) - Optional - Beam size for beam search.
- **patience** (float) - Optional - Beam search patience.
- **num_hypotheses** (int) - Optional - Number of hypotheses to return.
- **length_penalty** (float) - Optional - Length penalty applied to scores.
- **repetition_penalty** (float) - Optional - Penalty applied to the score of previously generated tokens (set > 1 to penalize).
- **no_repeat_ngram_size** (int) - Optional - Prevent repetitions of ngrams with this size (set 0 to disable).
- **disable_unk** (bool) - Optional - Disable the generation of the unknown token.
- **suppress_sequences** (Optional[List[List[str]]]) - Optional - Disable the generation of some sequences of tokens.
- **end_token** (Optional[Union[str, List[str], List[int]]]) - Optional - Stop the decoding on one of these tokens (defaults to the model EOS token).
- **return_end_token** (bool) - Optional - Whether to include the end token in the result.
- **max_length** (int) - Optional - Maximum generation length.
- **min_length** (int) - Optional - Minimum generation length.
- **static_prompt** (Optional[List[str]]) - Optional - If the model expects a static prompt (a.k.a. system prompt) it can be set here to simplify the inputs and optionally cache the model state for this prompt to accelerate future generations.
- **cache_static_prompt** (bool) - Optional - Cache the model state after the static prompt and reuse it for future generations using the same static prompt.
- **include_prompt_in_result** (bool) - Optional - Whether to include the prompt tokens in the result.
- **return_scores** (bool) - Optional - Whether to return scores.
- **return_logits_vocab** (bool) - Optional - Whether to return logits over the vocabulary.
- **return_alternatives** (bool) - Optional - Whether to return alternative hypotheses.
- **min_alternative_expansion_prob** (float) - Optional - Minimum probability for alternative expansion.
- **sampling_topk** (int) - Optional - Randomly sample predictions from the top K candidates.
- **sampling_topp** (float) - Optional - Keep the most probable tokens whose cumulative probability exceeds this value.
- **sampling_temperature** (float) - Optional - Sampling temperature to generate more random samples.
- **callback** (Callable[[GenerationStepResult], bool]) - Optional - Function called for each generated token when beam_size is 1. If it returns True, decoding stops for that batch index.

### Returns
- Union[List[GenerationResult], List[AsyncGenerationResult]] - A list of generation results, or an async generator iterator over `ctranslate2.GenerationStepResult` instances if asynchronous is True.

### Notes
- This generation method is not compatible with beam search which requires a complete decoding.
- The way the start tokens are forwarded in the decoder depends on `include_prompt_in_result`:
  - If `include_prompt_in_result` is True (default), the decoding loop is constrained to generate the start tokens that are then included in the result.
  - If `include_prompt_in_result` is False, the start tokens are forwarded in the decoder at once to initialize its state (i.e. the KV cache for Transformer models). For variable-length inputs, only the tokens up to the minimum length in the batch are forwarded at once. The remaining tokens are generated in the decoding loop with constrained decoding.
- Consider setting `include_prompt_in_result=False` to increase the performance for long inputs.
```

--------------------------------

### Generator.score_iterable

Source: https://opennmt.net/CTranslate2/python/ctranslate2.Generator.html

Scores an iterable of tokenized examples. This method is built on top of `ctranslate2.Generator.score_batch()` to efficiently score an arbitrarily large stream of data.

```APIDOC
## Generator.score_iterable

### Description
Scores an iterable of tokenized examples. This method is built on top of `ctranslate2.Generator.score_batch()` to efficiently score an arbitrarily large stream of data. It enables the following optimizations: stream processing, parallel scoring, asynchronous batch prefetching, and local sorting by length.

### Method
score_iterable

### Parameters
- **tokens** (Iterable[List[str]]) - Required - An iterable of tokenized examples.
- **max_batch_size** (int) - Optional - The maximum batch size.
- **batch_type** (str) - Optional - Whether `max_batch_size` is the number of "examples" or "tokens".
- **kwargs** - Optional - Any score options accepted by `ctranslate2.Generator.score_batch()`.

### Returns
A generator iterator over `ctranslate2.ScoringResult` instances.
```

--------------------------------

### OpenNMTTFConverter.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.OpenNMTTFConverter.html

Initializes the converter with an OpenNMT-tf model instance.

```APIDOC
## OpenNMTTFConverter.__init__

### Description
Initializes the converter with an OpenNMT-tf model instance.

### Method
Constructor

### Signature
`__init__(model)`

### Parameters
- **model** (opennmt.models.Model) - Required - An initialized and fully-built OpenNMT-tf model instance.

```

--------------------------------

### OpenNMTTFConverter.convert_from_args

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.OpenNMTTFConverter.html

Helper function to call the convert method with parsed command line options.

```APIDOC
## OpenNMTTFConverter.convert_from_args

### Description
Helper function to call `ctranslate2.converters.Converter.convert()` with the parsed command line options.

### Method
convert_from_args

### Signature
`convert_from_args(args: Namespace) -> str`

### Parameters
- **args** (Namespace) - Required - Namespace containing parsed arguments.

### Returns
- **str** - Path to the output directory.

```

--------------------------------

### OpenNMTTFConverter.from_config

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.OpenNMTTFConverter.html

Creates the converter from a configuration file or dictionary.

```APIDOC
## OpenNMTTFConverter.from_config

### Description
Creates the converter from the configuration.

### Method
from_config (classmethod)

### Signature
`from_config(config: Union[str, dict], auto_config: bool = False, checkpoint_path: Optional[str] = None, model: Optional[str] = None)`

### Parameters
- **config** (Union[str, dict]) - Required - Path to the YAML configuration, or a dictionary with the loaded configuration.
- **auto_config** (bool) - Optional - Whether the model automatic configuration values should be used.
- **checkpoint_path** (Optional[str]) - Optional - Path to the checkpoint or checkpoint directory to load. If not set, the latest checkpoint from the model directory is loaded.
- **model** (Optional[str]) - Optional - If the model instance cannot be resolved from the model directory, this argument can be set to either the name of the model in the catalog or the path to the model configuration.

### Returns
- **ctranslate2.converters.OpenNMTTFConverter** - An instance of the converter.

```

--------------------------------

### Download and convert WMT16 English-German model

Source: https://opennmt.net/CTranslate2/guides/fairseq.html

Download and convert the pretrained WMT16 English-German Transformer model. The output directory is used in the subsequent translation example.

```shell
wget https://dl.fbaipublicfiles.com/fairseq/models/wmt16.en-de.joined-dict.transformer.tar.bz2
tar xf wmt16.en-de.joined-dict.transformer.tar.bz2

ct2-fairseq-converter --model_path wmt16.en-de.joined-dict.transformer/model.pt \
    --data_dir wmt16.en-de.joined-dict.transformer \
    --output_dir ende_ctranslate2

```

--------------------------------

### ModelSpec.__init__()

Source: https://opennmt.net/CTranslate2/python/ctranslate2.specs.ModelSpec.html

Initializes the model specification.

```APIDOC
## ModelSpec.__init__()

### Description
Initializes the model specification.

### Method
Constructor

### Signature
`__init__()`

### Parameters
None

### Returns
None

```

--------------------------------

### Device.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.Device.html

Constructor for the Device enum, accepting an integer value representing the device type.

```APIDOC
## Device.__init__

### Description
Initializes a Device instance with an integer value.

### Method
Constructor

### Signature
`__init__(_value: int) -> None`

### Parameters
- **value** (int) - Required - The integer value representing the device type.

### Returns
None
```

--------------------------------

### Instantiate and forward a batch with ctranslate2.Encoder

Source: https://opennmt.net/CTranslate2/python/ctranslate2.Encoder.html

Creates an encoder on the CPU and forwards a batch of tokenized sequences. The example shows the expected usage pattern for the Encoder class.

```python
>>> encoder = ctranslate2.Encoder("model/", device="cpu")
>>> encoder.forward_batch([["▁Hello", "▁world", "!"]])
```

--------------------------------

### Get supported compute types for CPU and CUDA

Source: https://opennmt.net/CTranslate2/python/ctranslate2.get_supported_compute_types.html

Shows the supported compute types for CPU and CUDA devices. The output sets are returned by the function for each device.

```python
>>> ctranslate2.get_supported_compute_types("cpu")
{'int16', 'float32', 'int8', 'int8_float32'}
>>> ctranslate2.get_supported_compute_types("cuda")
{'float32', 'int8_float16', 'float16', 'int8', 'int8_float32'}

```

--------------------------------

### Generator.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.Generator.html

Initializes the generator with a model path and various configuration options.

```APIDOC
## Generator.__init__

### Description
Initializes the generator with a model path and various configuration options.

### Method
Constructor

### Signature
`__init__(model_path: str, device: str = 'cpu', *, device_index: Union[int, List[int]] = 0, compute_type: Union[str, Dict[str, str]] = 'default', inter_threads: int = 1, intra_threads: int = 0, max_queued_batches: int = 0, flash_attention: bool = False, tensor_parallel: bool = False, files: object = None) -> None`

### Parameters
- **model_path** (str) - Required - Path to the CTranslate2 model directory.
- **device** (str) - Optional - Device to use (possible values are: cpu, cuda, auto).
- **device_index** (Union[int, List[int]]) - Optional - Device IDs where to place this generator on.
- **compute_type** (Union[str, Dict[str, str]]) - Optional - Model computation type or a dictionary mapping a device name to the computation type (possible values are: default, auto, int8, int8_float32, int8_float16, int8_bfloat16, int16, float16, bfloat16, float32).
- **inter_threads** (int) - Optional - Maximum number of parallel generations.
- **intra_threads** (int) - Optional - Number of OpenMP threads per generator (0 to use a default value).
- **max_queued_batches** (int) - Optional - Maximum numbers of batches in the queue (-1 for unlimited, 0 for an automatic value). When the queue is full, future requests will block until a free slot is available.
- **flash_attention** (bool) - Optional - run model with flash attention 2 for self-attention layer
- **tensor_parallel** (bool) - Optional - run model with tensor parallel mode.
- **files** (object) - Optional - Load model files from the memory. This argument is a dictionary mapping file names to file contents as file-like or bytes objects. If this is set, `model_path` acts as an identifier for this model.

### Returns
None
```

--------------------------------

### Convert and translate with t5-small

Source: https://opennmt.net/CTranslate2/guides/transformers.html

Convert the t5-small model with ct2-transformers-converter, then load it with ctranslate2.Translator and translate a machine translation input using the AutoTokenizer. The example prints the decoded translation.

```bash
ct2-transformers-converter --model t5-small --output_dir t5-small-ct2

```

```python
import ctranslate2
import transformers

translator = ctranslate2.Translator("t5-small-ct2")
tokenizer = transformers.AutoTokenizer.from_pretrained("t5-small")

input_text = "translate English to German: The house is wonderful."
input_tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(input_text))

results = translator.translate_batch([input_tokens])

output_tokens = results[0].hypotheses[0]
output_text = tokenizer.decode(tokenizer.convert_tokens_to_ids(output_tokens))

print(output_text)

```

--------------------------------

### MarianConverter.__init__

Source: https://opennmt.net/CTranslate2/python/ctranslate2.converters.MarianConverter.html

Initializes the Marian converter with the model and vocabulary paths.

```APIDOC
## MarianConverter.__init__

### Description
Initializes the Marian converter with the model and vocabulary paths.

### Method
Constructor

### Signature
`__init__(model_path: str, vocab_paths: List[str])`

### Parameters
- **model_path** (str) - Required - Path to the Marian model (.npz file).
- **vocab_paths** (List[str]) - Required - Paths to the vocabularies (.yml files).

### Returns
None
```
Tokens:
Raw