### Install uv Package Manager (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/README.md

Installs the `uv` package manager using pip. `uv` is a fast Python package installer and resolver.

```Shell
pip install uv
```

--------------------------------

### Setup Imports and Path

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/test_prepare_dataset.ipynb

Imports necessary libraries like `tokenizers`, `pathlib`, `sys`, and a local helper module `sentence_destructor`. It also modifies the system path to include the parent directory to resolve local module imports.

```Python
from tokenizers import Tokenizer
from pathlib import Path
import sys

sys.path.append(str(Path(Path.cwd()).parent))

from helper.sentence_destructor import destruct_sentence
```

--------------------------------

### Loading and Using Trained Tokenizer with tokenizers (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/generate_output_tokenizer.ipynb

This snippet shows how to load a previously trained tokenizer from a file using the `tokenizers` library. It sets a decoder, encodes an example string containing mixed English and Uyghur text, prints the resulting tokens and their corresponding IDs, and then decodes the IDs back into a string, demonstrating the tokenizer's ability to process and reconstruct text.

```python
from tokenizers import Tokenizer, decoders, Encoding

tokenizer: Tokenizer = Tokenizer.from_file("../../config/output_tokenizer.json")
tokenizer.decoder = decoders.Metaspace()

example_text = "<PAD><UNK>ئىنسانلار ئادەملەر بىلەن ئادەم، قاسىم ھاشىمغا ‹ياخشىمۇسىز» دىدى. ھاشىم: مەن ياخشى، ئۆزىڭىزچۇ؟ دېدى. يەنە باشقا مىساللاردىن گۈزەللىك، ياخشىلىق، ئەسكىلىك، ئايدىڭلاشتۇرالمايۋاتقانلىقىمىزدىنمىكىنتاڭHello"
tokenized: Encoding = tokenizer.encode(example_text)
print(tokenized.tokens)
print(tokenized.ids)
print(f"char length: {len(example_text)} vs token ids: {len(tokenized.ids)}")
decoded: str = tokenizer.decode(tokenized.ids, skip_special_tokens=True)
print("Source versus decoded:")
print(example_text)
print(decoded)
```

--------------------------------

### Install Project Dependencies with uv (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/README.md

Uses the `uv` package manager to synchronize project dependencies based on the project's lock file (e.g., `requirements.txt` or `pyproject.toml`). This command installs or updates packages to match the specified versions.

```Shell
uv sync
```

--------------------------------

### Test Dataset Sentence Destruction

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/test_prepare_dataset.ipynb

Loads a tokenizer from a file, defines a sample sentence in Uyghur, and repeatedly calls `destruct_sentence` with the tokenizer and sentence, printing the result. This tests the sentence destruction logic with a specific language example.

```Python
tokenizer: Tokenizer = Tokenizer.from_file("../../config/input_tokenizer.json")

text = "مەن مەكتەپتىن قايتىپ كەلدىم، ئۇمۇ مەكتەپتىن قايتىل كەلدى، بىز ھەممىمىز مەكتەپتىن قايتىپ كەلدۇق!"

for i in range(10):
    destructed = destruct_sentence(tokenizer, text, 0.15)

    print(destructed)
print(text)
```

--------------------------------

### Training BPE Tokenizer with tokenizers (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/generate_output_tokenizer.ipynb

This snippet demonstrates how to train a Byte Pair Encoding (BPE) tokenizer using the `tokenizers` library in Python. It initializes the tokenizer and trainer, specifies special tokens, loads text files from a directory, trains the tokenizer on these files, and saves the resulting tokenizer configuration to a JSON file.

```python
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, Encoding, decoders
from pathlib import Path

tokenizer = Tokenizer(models.BPE(unk_token="<UNK>", ignore_merges=False))
tokenizer.pre_tokenizer = pre_tokenizers.Metaspace()

trainer = trainers.BpeTrainer(special_tokens=["<PAD>", "<UNK>", "<SOS>", "<EOS>"], vocab_size=4096)

files = [str(i.absolute().resolve()) for i in Path("../../.data/text").iterdir() if i.is_file() and i.name.endswith(".json")]
print(f"files to be included: {files}")
tokenizer.train(files, trainer)

tokenizer.save("../../config/output_tokenizer.json")
print("Done.")
```

--------------------------------

### Load Tokenizer from File in Python

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/generate_input_tokenizer.ipynb

This snippet shows how to load a pre-configured tokenizer from a JSON file using the `Tokenizer.from_file` method provided by the Hugging Face `tokenizers` library.

```python
tokenizer: Tokenizer = Tokenizer.from_file("../../config/input_tokenizer.json")
```

--------------------------------

### Create, Configure, and Save CharBPETokenizer in Python

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/generate_input_tokenizer.ipynb

This snippet demonstrates how to initialize a `CharBPETokenizer`, load a custom vocabulary from a JSON file, add special tokens and the vocabulary, and save the tokenizer configuration to a JSON file. It also shows how to manually update the vocabulary within the saved JSON file.

```python
from tokenizers import Tokenizer, models, Encoding, pre_tokenizers, decoders, Regex, CharBPETokenizer, trainers
import json


with open("../../config/symbols.json", "r") as f:
    vocab_list: list[str] = json.loads(f.read())
special_tokens = ["<PAD>", "<UNK>", "<SOS>", "<EOS>"]
tokenizer: Tokenizer = CharBPETokenizer(unk_token="<UNK>")

tokenizer.add_special_tokens(special_tokens)
tokenizer.add_tokens(special_tokens)
tokenizer.add_tokens(vocab_list)

trainer = trainers.WordLevelTrainer(special_tokens=special_tokens)

tokenizer.save("../../config/input_tokenizer.json")
with open("../../config/input_tokenizer.json", "r") as f:
    obj = json.loads(f.read())
    v: dict[str, int] = tokenizer.get_vocab()
    obj["model"]["vocab"] = v
with open("../../config/input_tokenizer.json", "w") as f:
    f.write(json.dumps(obj, ensure_ascii=False, indent=2))
```

--------------------------------

### Demonstrate RegressiveBufferGenerator (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/sentence_length_statistics.ipynb

Imports components from `train.py`, creates sample token data, initializes a zero tensor buffer, copies tokens into the buffer using `copy_unpadded_tokens_into_buffer`, initializes a `RegressiveBufferGenerator`, and demonstrates generating data into another buffer (`y`) in a loop, printing the results.

```python
from train import RegressiveBufferGenerator, copy_unpadded_tokens_into_buffer
import torch
import functools
import torch
import functools

tokens = [
    [1, 2, 3, 4],
    [5, 6, 7, 8, 9],
    [10, 11]
]
x = torch.zeros(3, 5, dtype=torch.int32)
lengths = list(map(lambda i: len(i), tokens))
print("tokens", tokens)
# print("x before update\n",x)
copy_unpadded_tokens_into_buffer(tokens, x)
# print("x after update\n", x)
print("lengths:", lengths)
autoregressive_generator = RegressiveBufferGenerator(x, lengths)
print("count of generatable items", len(autoregressive_generator))
y = torch.zeros(5, x.shape[1], dtype=torch.int32)

for i in range(5):
    print(f"iter: {i}------------------------\n")
    copy_count = autoregressive_generator.generate_into(y, 5)
    print(f"copy count: {copy_count}")
    print(y)
    y.zero_()
```

--------------------------------

### Encode and Decode Text with Tokenizer in Python

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/generate_input_tokenizer.ipynb

This snippet demonstrates how to use a loaded tokenizer to encode a sample text string into token IDs and tokens, print the results, and then decode the token IDs back into a string to verify the process.

```python
text = "<EOS><SOS><PAD> Helloياخشىمۇسىز، قانداق ئەھۋالىڭىز؟ ئايدىڭلاشتۇرالمايۋاتقانلىقىڭىزدىنمىكىنتاڭ ئايدىڭلاشتۇرالمايۋاتقاندەك قىلىسىز."
encoded: Encoding = tokenizer.encode(text)
print(f"ids: {encoded.ids}")
print(f"encoded: {encoded.tokens}")
decoded: str = tokenizer.decode(encoded.ids)
print(f"len of tokens: {len(encoded.ids)}")
print(f"Source : {text}")
print(f"Decoded: {decoded}")
```

--------------------------------

### Load Dataset and Tokenizers (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/sentence_length_statistics.ipynb

Imports necessary libraries, loads input and output tokenizers from files, initializes a SentenceDataset with a specified file and tokenizers, and prints statistics about the loaded dataset (sentence counts and max lengths).

```python
from pathlib import Path
import json
from helper.sentence_destructor import destruct_sentence
import random
from tokenizers import Tokenizer, Encoding
import tqdm
from typing import Sequence
from helper.dataset import SentenceDataset

input_tokenizer: Tokenizer = Tokenizer.from_file(str(Path.cwd().parent / "config/input_tokenizer.json"))
output_tokenizer: Tokenizer = Tokenizer.from_file(str(Path.cwd().parent / "config/output_tokenizer.json"))

print(input_tokenizer.encode("<UNK><EOS><SOS><PAD>hello").ids)
data_dir = Path.cwd().parent / ".data"
dataset_file = data_dir / "text" / "2000.json"
my_dataset = SentenceDataset(dataset_file, input_tokenizer, output_tokenizer, broken_sentence_variation_count=2)


print(f"sentence length (correct)  {len(my_dataset.correct_sentences)}")
print(f"sentence length (broken )  {len(my_dataset.broken_sentences)}")

print(f"max sentence length (correct) {my_dataset.max_correct_sentence_length}")
print(f"max sentence length (broken ) {my_dataset.max_broken_sentence_length}")
```

--------------------------------

### Benchmark GPU Memory Copy Overhead

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/test_prepare_dataset.ipynb

Performs a benchmark to measure the time taken to copy data between tensors on the GPU. It creates dummy tensors and a buffer on the CUDA device and repeatedly copies data into the buffer 200 times (100 iterations * 2 copies), measuring the elapsed time in milliseconds.

```Python
import torch
import time

my_device = "cuda:0"

dummy_data1 = torch.randint(-32768, 32767, (1024, 2048, 128), dtype=torch.int32, device=my_device)
dummy_data2 = torch.randint(-32768, 32767,(1024, 2048, 128), dtype=torch.int32, device=my_device)

my_buffer = torch.zeros((1024, 2048, 128), dtype=torch.int32, device=my_device)

begin = time.time()
for i in range(100):
    my_buffer[:, :, :] = dummy_data1[:, :, :]
    my_buffer[:, :, :] = dummy_data2[:, :, :]
elapsed_ms = int((time.time() - begin) * 100)

del dummy_data1, dummy_data2, my_buffer
torch.cuda.empty_cache()

print(f"time to taken by copy data to buffer 2000 times: {elapsed_ms}")
```

--------------------------------

### Plot Correct Sentence Length Distribution (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/sentence_length_statistics.ipynb

Uses the `seaborn` library to create a line plot visualizing the distribution of sentence lengths for the correct sentences in the dataset, using the `calculate_statistics` function. Sets labels and title for the plot.

```python
a = seaborn.lineplot(data=calculate_statistics(my_dataset.correct_sentences))
a.set_xlabel("sentence length")
a.set_ylabel("sentence count")
a.set_title("Correct sentence count")
```

--------------------------------

### Plot Broken Sentence Length Distribution (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/sentence_length_statistics.ipynb

Uses the `seaborn` library to create a line plot visualizing the distribution of sentence lengths for the broken sentences in the dataset, using the `calculate_statistics` function and mapping the broken sentences to their token lists. Sets labels and title for the plot.

```python
b = seaborn.lineplot(data=calculate_statistics(map(lambda item: item[0], my_dataset.broken_sentences)))
b.set_xlabel("sentence length")
b.set_ylabel("sentence count")
b.set_title("Broken sentence count")
```

--------------------------------

### Write Processed Sentences to File

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/test_prepare_dataset.ipynb

Writes the `expanded_sentences` list (processed in the previous snippet) to a JSON file named `dataset.json` in the data directory. The `ensure_ascii=False` flag is used to preserve non-ASCII characters.

```Python
with open(data_dir / "dataset.json", "w+") as file:
    file.write(json.dumps(expanded_sentences, ensure_ascii=False))
print("[OK] of write shirnked sentence to file.")
```

--------------------------------

### Calculate Sentence Length Statistics (Python)

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/sentence_length_statistics.ipynb

Defines a Python function `calculate_statistics` that takes a list of integer lists (representing tokenized sentences) and returns a dictionary where keys are sentence lengths and values are the count of sentences with that length.

```python
import seaborn
from typing import Sequence

def calculate_statistics(items: list[list[int]]) -> dict[int, int]:
    lengths: dict[int, int] = {}

    for s in items:
        length = len(s)
        lengths[length] = lengths[length] + 1 if length in lengths else 1
    return lengths
```

--------------------------------

### Split and Filter Sentences by Length

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/test_prepare_dataset.ipynb

Reads sentences from a JSON file, splits them into sub-sentences based on dots ('.'), filters out empty or short sub-sentences (length <= 15), and prints the counts before and after splitting, as well as the maximum resulting sentence length.

```Python
import json

data_dir = Path(Path.cwd()).parent.parent / ".data"
dataset_file = data_dir / "text" / "2000.json"
with open(dataset_file, "r") as file:
    sentences: list[str] = json.loads(file.read())
print(f"sentence count before split: {len(sentences)}")
expanded_sentences: list[str] = []

for s in sentences:
    ss = [ss for ss in s.strip().split(".") if not (ss.strip() == "")]
    ss = [s for s in ss if len(s) > 15]
    expanded_sentences.extend(ss)
print(f"sentence count after split: {len(expanded_sentences)}")

max_sentence_length = 0
for s in expanded_sentences:
    max_sentence_length = max(max_sentence_length, len(s))

print(f"max sentence length after split: {max_sentence_length}")
```

--------------------------------

### Allocate GPU Memory for Training Data

Source: https://github.com/dream-1ab/spell_corrector/blob/main/src/tools/test_prepare_dataset.ipynb

Allocates a PyTorch tensor on the CUDA device (`cuda:0`) to hold training data. The tensor size is determined by the number of processed sentences (`len(expanded_sentences)`) and the maximum sentence length (`max_sentence_length`) found in previous steps.

```Python
import torch
training_data = torch.zeros(len(expanded_sentences), max_sentence_length, dtype=torch.int32, device="cuda:0")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.