### Install pytest and Run Tests Source: https://github.com/vkcom/youtokentome/blob/master/tests/unit_tests/README.md Installs the pytest framework and then executes all unit tests for the project. Testing may take several minutes. ```bash pip install pytest pytest ``` -------------------------------- ### Install YouTokenToMe Source: https://github.com/vkcom/youtokentome/blob/master/README.md Install the library using pip. ```bash pip install youtokentome ``` -------------------------------- ### Python Interface Example Source: https://github.com/vkcom/youtokentome/blob/master/README.md Demonstrates training a BPE model, loading it, and performing tokenization in two different output formats (ID and SUBWORD). ```python import random import youtokentome as yttm train_data_path = "train_data.txt" model_path = "example.model" # Generating random file with training data # 10000 lines with 100 characters in each line n_lines = 10000 n_characters = 100 with open(train_data_path, "w") as fout: for _ in range(n_lines): print("".join([random.choice("abcd ") for _ in range(n_characters)]), file=fout) # Generating random text test_text = "".join([random.choice("abcde ") for _ in range(100)]) # Training model yttm.BPE.train(data=train_data_path, vocab_size=5000, model=model_path) # Loading model bpe = yttm.BPE(model=model_path) # Two types of tokenization print(bpe.encode([test_text], output_type=yttm.OutputType.ID)) print(bpe.encode([test_text], output_type=yttm.OutputType.SUBWORD)) ``` -------------------------------- ### Get Vocabulary Size Source: https://github.com/vkcom/youtokentome/blob/master/README.md Returns the total number of unique subwords in the vocabulary. ```python vocab_size(self) ``` -------------------------------- ### vocab_size Source: https://github.com/vkcom/youtokentome/blob/master/README.md Gets the total number of unique subwords in the tokenizer's vocabulary. ```APIDOC ## vocab_size ### Description Returns the total number of unique subwords available in the tokenizer's vocabulary. ### Method Signature `vocab_size(self)` ### Returns * An integer representing the size of the vocabulary. ``` -------------------------------- ### Get Vocabulary List Source: https://github.com/vkcom/youtokentome/blob/master/README.md Retrieves the list of all subwords present in the vocabulary. The order of subwords in the list corresponds to their respective IDs. ```python vocab(self) ``` -------------------------------- ### Get Subword from ID Source: https://github.com/vkcom/youtokentome/blob/master/README.md Retrieves the subword corresponding to a given integer ID. The ID must be within the valid vocabulary range. ```python id_to_subword(self, id) ``` -------------------------------- ### Get ID for a Subword Source: https://github.com/vkcom/youtokentome/blob/master/README.md Finds the integer ID associated with a given subword. If the subword is not in the vocabulary, the unknown ID (`unk_id`) is returned. ```python subword_to_id(self, subword) ``` -------------------------------- ### Build and Run Benchmark with Docker Source: https://github.com/vkcom/youtokentome/blob/master/tests/speed_test/README.md Builds the Docker image for the speed test and runs it, mounting a local directory for data storage. Ensure PATH_TO_DOWNLOADED_DATA is an absolute path. ```bash cd tests/speed_test docker build -t yttm/speed_test . docker run --rm -v PATH_TO_DOWNLOADED_DATA:/workspace/data -it yttm/speed_test:latest ``` -------------------------------- ### YouTokenToMe BPE Training Help Source: https://github.com/vkcom/youtokentome/blob/master/README.md Shows the help message for the 'bpe' command, detailing options for training a Byte Pair Encoding model. ```bash $ yttm bpe --help Usage: yttm bpe [OPTIONS] Train BPE model. Options: --data PATH Training data file path. [required] --model PATH Output model file path. [required] --vocab_size INTEGER Number of tokens in the final vocabulary. [required] --coverage FLOAT Fraction of characters covered by the model. [default: 1.0] --n_threads INTEGER Number of threads. [default: -1] --pad_id INTEGER Padding token id. [default: 0] --unk_id INTEGER Unknown token id. [default: 1] --bos_id INTEGER 'Begin of sentence' token id. [default: 2] --eos_id INTEGER 'End of sentence' token id. [default: 3] --help Show this message and exit. ``` -------------------------------- ### Train BPE Model via CLI Source: https://github.com/vkcom/youtokentome/blob/master/README.md Trains a BPE model using the command-line interface. Requires specifying training data and an output model file, along with the desired vocabulary size. ```bash yttm bpe --data TRAINING_DATA_FILE --model OUTPUT_MODEL_FILE --vocab_size 2000 ``` -------------------------------- ### YouTokenToMe CLI Help Source: https://github.com/vkcom/youtokentome/blob/master/README.md Displays the main help message for the YouTokenToMe CLI, listing available commands and general options. ```bash $ yttm --help Usage: yttm [OPTIONS] COMMAND [ARGS]... Options: --help Show this message and exit. Commands: bpe Train BPE model. decode Decode ids to text. encode Encode text to ids or subwords. vocab Print list of learned subwords. ``` -------------------------------- ### BPE Constructor Source: https://github.com/vkcom/youtokentome/blob/master/README.md Loads a pre-trained BPE model from a specified file path. It allows configuration of the number of threads to be used for loading and processing. ```APIDOC ## BPE(model, n_threads=-1) Class constructor. Loads the trained model. ### Parameters * `model` (string) - Required - path to the trained model * `n_threads` (int) - Optional - number of parallel threads used to run. If equal to -1, then the maximum number of threads available will be used. ``` -------------------------------- ### YouTokenToMe Encoding Help Source: https://github.com/vkcom/youtokentome/blob/master/README.md Displays the help message for the 'encode' command, outlining options for converting text to IDs or subwords. ```bash $ yttm encode --help Usage: yttm encode [OPTIONS] Encode text to ids or subwords. Options: --model PATH Path to file with learned model. [required] --output_type TEXT 'id' or 'subword'. [required] --n_threads INTEGER Number of threads. [default: -1] --bos Add tab 'begin of sentence'. --eos Add tab 'end of sentence'. --reverse Reverse output sequence of tokens. --stream Process each line before reading the next one. --dropout_prob BPE-dropout probability (the probability of a merge being dropped). [default: 0] --help Show this message and exit. ``` -------------------------------- ### YouTokenToMe Vocabulary Help Source: https://github.com/vkcom/youtokentome/blob/master/README.md Shows the help message for the 'vocab' command, which is used to print the learned subwords from a model. ```bash $ yttm vocab --help Usage: yttm vocab [OPTIONS] Print list of learned subwords. Options: --model PATH Path to file with learned model. [required] --verbose Add merging rules. --help Show this message and exit. ``` -------------------------------- ### Load BPE Model Source: https://github.com/vkcom/youtokentome/blob/master/README.md Loads a trained BPE model from a specified file path. The number of threads can also be configured. ```python youtokentome.BPE(model, n_threads=-1) ``` -------------------------------- ### YouTokenToMe Decoding Help Source: https://github.com/vkcom/youtokentome/blob/master/README.md Displays the help message for the 'decode' command, detailing options for converting IDs back to text. ```bash $ yttm decode --help Usage: yttm decode [OPTIONS] Decode ids to text. Options: --model PATH Path to file with learned model. [required] --ignore_ids List of indices to ignore for decoding. Example: --ignore_ids=1,2,3 --help Show this message and exit. ``` -------------------------------- ### Train BPE Model Source: https://github.com/vkcom/youtokentome/blob/master/README.md Trains a BPE model and saves it to a file. Specify data path, model save path, vocabulary size, and coverage. Optional parameters include number of threads and special token IDs. ```python youtokentome.BPE.train(data, model, vocab_size, coverage, n_threads=-1, pad_id=0, unk_id=1, bos_id=2, eos_id=3) ``` -------------------------------- ### BPE.train Source: https://github.com/vkcom/youtokentome/blob/master/README.md Trains a BPE model using the provided data and saves it to a specified file. It allows configuration of vocabulary size, character coverage, and threading. It returns a loaded BPE model class. ```APIDOC ## BPE.train Trains BPE model and saves to file. ### Method Signature ```python youtokentome.BPE.train(data, model, vocab_size, coverage, n_threads=-1, pad_id=0, unk_id=1, bos_id=2, eos_id=3) ``` ### Parameters * `data` (string) - Required - path to file with training data * `model` (string) - Required - path to where the trained model will be saved * `vocab_size` (int) - Required - number of tokens in the final vocabulary * `coverage` (float) - Required - fraction of characters covered by the model. Must be in the range [0, 1]. A good value to use is about 0.9999. * `n_threads` (int) - Optional - number of parallel threads used to run. If -1 is passed, then all available threads are going to be used. Note that the number of threads is limited by 8. * `pad_id` (int) - Optional - reserved id for padding * `unk_id` (int) - Optional - reserved id for unknown symbols * `bos_id` (int) - Optional - reserved id for begin of sentence token * `eos_id` (int) - Optional - reserved id for end of sentence token ### Returns Class `youtokentome.BPE` with the loaded model. ``` -------------------------------- ### Encode Text using CLI Source: https://github.com/vkcom/youtokentome/blob/master/README.md Encodes input data using a pre-trained model via the command-line interface. Allows specifying the output type and redirects input/output streams. ```bash yttm encode --model OUTPUT_MODEL_FILE --output_type subword < TEST_DATA_FILE > ENCODED_DATA ``` -------------------------------- ### vocab Source: https://github.com/vkcom/youtokentome/blob/master/README.md Retrieves the vocabulary of the tokenizer as a list of subword strings. ```APIDOC ## vocab ### Description Returns the entire vocabulary of the tokenizer as a list of strings, where each string is a subword. ### Method Signature `vocab(self)` ### Returns * A list of strings, representing all subwords in the vocabulary. The size of the list is equal to `vocab_size`. ``` -------------------------------- ### decode Source: https://github.com/vkcom/youtokentome/blob/master/README.md Converts a list of token IDs back into a list of human-readable strings. ```APIDOC ## decode ### Description Converts a list of token IDs back into their corresponding subword strings and concatenates them with spaces. It can optionally ignore specific IDs during the decoding process. ### Method Signature `decode(self, ids, ignore_ids=None)` ### Parameters * `ids` (list of lists of integers) - Required - A list where each element is a list of token IDs to be decoded. All integers must be in the range [0, vocab_size-1]. * `ignore_ids` (collection of integers) - Optional - A collection of IDs that should be ignored during the decoding process. Defaults to None. ### Returns * A list of strings, where each string is the decoded sequence of subwords. ``` -------------------------------- ### Encode Sentences to IDs or Subwords Source: https://github.com/vkcom/youtokentome/blob/master/README.md Tokenizes a list of sentences. Supports options for adding sentence boundaries, reversing sequences, and applying BPE dropout. The output type can be specified as IDs or subwords. ```python encode(self, sentences, output_type=yttm.OutputType.ID, bos=False, eos=False, reverse=False, dropout_prob=0) ``` -------------------------------- ### id_to_subword Source: https://github.com/vkcom/youtokentome/blob/master/README.md Converts a given integer ID into its corresponding subword string. ```APIDOC ## id_to_subword ### Description Retrieves the subword string associated with a given integer ID from the vocabulary. ### Method Signature `id_to_subword(self, id)` ### Parameters * `id` (int) - Required - The integer ID of the subword. Must be in the range [0, vocab_size-1]. ### Returns * A string representing the subword corresponding to the provided ID. ``` -------------------------------- ### encode Source: https://github.com/vkcom/youtokentome/blob/master/README.md Tokenizes a list of sentences into either IDs or subwords. Supports options for adding sentence boundaries, reversing sequences, and applying dropout. ```APIDOC ## encode ### Description Tokenizes a list of sentences into numerical IDs or subword strings. This method allows for customization of the output format and includes options for sentence boundary tokens, sequence reversal, and dropout. ### Method Signature `encode(self, sentences, output_type=yttm.OutputType.ID, bos=False, eos=False, reverse=False, dropout_prob=0)` ### Parameters * `sentences` (list of strings) - Required - The list of sentences to be tokenized. * `output_type` (enum) - Optional - Specifies the type of output. Use `OutputType.ID` for token IDs or `OutputType.SUBWORD` for subword strings. Defaults to `OutputType.ID`. * `bos` (bool) - Optional - If True, adds a 'beginning of sentence' token to the output. Defaults to False. * `eos` (bool) - Optional - If True, adds an 'end of sentence' token to the output. Defaults to False. * `reverse` (bool) - Optional - If True, reverses the order of the output token sequence. Defaults to False. * `dropout_prob` (float) - Optional - The probability of dropping a merge operation during BPE tokenization. Must be between 0 and 1. Defaults to 0. ### Returns * A list of lists of integers if `output_type` is `OutputType.ID`. * A list of lists of strings if `output_type` is `OutputType.SUBWORD`. ``` -------------------------------- ### Decode IDs to Subwords Source: https://github.com/vkcom/youtokentome/blob/master/README.md Converts a list of token IDs back into a list of subword strings, concatenating them with spaces. Specific IDs can be ignored during the decoding process. ```python decode(self, ids, ignore_ids=None) ``` -------------------------------- ### subword_to_id Source: https://github.com/vkcom/youtokentome/blob/master/README.md Converts a given subword string into its corresponding integer ID. ```APIDOC ## subword_to_id ### Description Maps a subword string to its unique integer identifier within the vocabulary. If the subword is not found, the unknown token ID (`unk_id`) is returned. ### Method Signature `subword_to_id(self, subword)` ### Parameters * `subword` (string) - Required - The subword to convert to an ID. ### Returns * An integer representing the ID of the subword, within the range [0, vocab_size-1]. Returns `unk_id` if the subword is not in the vocabulary. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.