### Install Dependencies

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Installs essential Python libraries for NLP tasks, including sentencepiece, transformers, datasets, torch, scikit-learn, and nepalitokenizers.

```python
%pip install sentencepiece transformers datasets torch scikit-learn
%pip install nepalitokenizers
```

--------------------------------

### Install tiktoken

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Installs the tiktoken library, which is used for BPE tokenization.

```Python
! pip3 install tiktoken
```

--------------------------------

### Install Unsloth

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Installs the Unsloth library, including the latest nightly version from GitHub. This library is optimized for faster LLM training.

```Shell
%%capture
!pip install unsloth
# Also get the latest nightly Unsloth!
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git
```

--------------------------------

### Install tqdm Package

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

This command installs the 'tqdm' Python package, which is commonly used for displaying progress bars in loops and long-running processes.

```shell
%pip install tqdm
```

--------------------------------

### Install PyTorch

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Installs PyTorch with CUDA support for the specified version (cu118). This is a prerequisite for using GPU acceleration.

```Shell
%pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
```

--------------------------------

### PyTorch RNN Language Model Setup

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

This Python script sets up a PyTorch environment for training a language model. It loads tokenized data and Word2Vec embeddings, defines a `SmallTextGenerator` class using LSTM, and prepares data batches for training.

```python
import torch
import torch.nn as nn
import torch.optim as optim
import pickle
import numpy as np
from torch.nn.utils.rnn import pad_sequence
from tqdm import tqdm  # Importing tqdm for progress bar

# Enable anomaly detection for debugging in-place errors
torch.autograd.set_detect_anomaly(True)

# Load tokenized data and Word2Vec embeddings
with open('wordpiece_tokenized.pkl', 'rb') as f:
    tokenized_data = pickle.load(f)

# Ensure that we have 'input_ids' and 'tokens' in the data
input_ids = tokenized_data.get('input_ids', [])
tokens = tokenized_data.get('tokens', [])

if not input_ids or not tokens:
    raise ValueError("The tokenized data does not contain 'input_ids' or 'tokens'.")

# Load Word2Vec embeddings
word2vec_path = "processed_normalized.word2vec.wv.vectors.npy"
word2vec = np.load(word2vec_path)

# Create embedding matrix
vocab_file = 'wordpiece_vocab.vocab'

# Read the vocabulary file and create a mapping
vocab = {}
idx_to_word = {}
with open(vocab_file, 'r', encoding='utf-8') as f:
    for idx, line in enumerate(f):
        token = line.strip()  # Remove any extra whitespace
        vocab[token] = idx
        idx_to_word[idx] = token

# Get the vocabulary size
vocab_size = len(vocab)
print(f"Vocabulary Size: {vocab_size}")

embedding_dim = word2vec.shape[1]
embedding_matrix = torch.tensor(word2vec, dtype=torch.float32)

# Model: Simple RNN-based Language Model
class SmallTextGenerator(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, num_layers):
        super(SmallTextGenerator, self).__init__()
        self.embedding = nn.Embedding.from_pretrained(embedding_matrix, freeze=False)
        self.rnn = nn.LSTM(embedding_dim, hidden_dim, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, vocab_size)

    def forward(self, x, hidden=None):
        embeds = self.embedding(x)
        output, hidden = self.rnn(embeds, hidden)
        logits = self.fc(output)
        return logits, hidden

# Hyperparameters
hidden_dim = 128
num_layers = 2
learning_rate = 0.001
batch_size = 32
num_epochs = 10

# Initialize model, loss, and optimizer
model = SmallTextGenerator(vocab_size, embedding_dim, hidden_dim, num_layers)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

# Prepare data for training
def get_batches(data, batch_size):
    for i in range(0, len(data) - batch_size + 1, batch_size):
        batch = data[i:i+batch_size]
        # Pad sequences to the same length within a batch
        padded_batch = pad_sequence([torch.tensor(seq, dtype=torch.long) for seq in batch], batch_first=True)
        yield padded_batch

```

--------------------------------

### Execute Model Training

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Starts the training process using the configured UnslothTrainer. The `train()` method executes the training loop and returns statistics upon completion.

```Python
trainer_stats = trainer.train()

```

--------------------------------

### Initialize tiktoken tokenizer

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Imports the tiktoken library and retrieves the version. It then gets the 'gpt2' encoding for tokenization.

```Python
import importlib
import tiktoken

print("tiktoken version:", importlib.metadata.version("tiktoken"))

tokenizer = tiktoken.get_encoding("gpt2")
```

--------------------------------

### Check PyTorch and CUDA Availability

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Checks the installed PyTorch version and verifies if CUDA is available, which is necessary for GPU acceleration.

```Python
import torch
print(torch.__version__)  # Check PyTorch version
print(torch.cuda.is_available())  # Check if CUDA is available
```

--------------------------------

### Tokenize and Decode Example

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Demonstrates the usage of the SimpleTokenizerV2 by encoding a combined text string (containing two sentences separated by '<|endoftext|>') into token IDs and then decoding these IDs back into a human-readable string. This also highlights how unknown words are handled.

```python
tokenizer = SimpleTokenizerV2(vocab)

text1 = "Hello, do you like tea?"
text2 = "In the sunlit terraces of the palace."

text = " <|endoftext|> ".join((text1, text2))

print(text)

print(tokenizer.encode(text))

print(tokenizer.decode(tokenizer.encode(text)))
```

--------------------------------

### Get Initial GPU Memory Statistics

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Retrieves and prints the GPU name, total available memory, and the currently reserved memory before training begins. This helps in monitoring resource usage.

```Python
#@title Show current memory stats
gpu_stats = torch.cuda.get_device_properties(0)
start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.")
print(f"{start_gpu_memory} GB of memory reserved.")

```

--------------------------------

### Apply Token Embeddings

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Prints the weights of the token embedding layer and then applies the embedding layer to a tensor of token IDs to get their corresponding embeddings.

```Python
print(embedding_layer.weight)

print(embedding_layer(torch.tensor([3])))

print(embedding_layer(input_ids))
```

--------------------------------

### Get Tokenized Text Length

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Prints the total number of tokens after the preprocessing and tokenization steps have been applied to the text.

```Python
print(len(preprocessed))
```

--------------------------------

### Generate Positional Embeddings

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Creates positional embeddings for a sequence of a given context length. It generates a tensor of positional indices and then uses the embedding layer to get the corresponding embeddings.

```Python
max_length = 4
dataloader = create_dataloader_v1(
    raw_text, batch_size=8, max_length=max_length,
    stride=max_length, shuffle=False
)
data_iter = iter(dataloader)
inputs, targets = next(data_iter)

print("Token IDs:\n", inputs)
print("\nInputs shape:\n", inputs.shape)

token_embeddings = token_embedding_layer(inputs)
print(token_embeddings.shape)

context_length = max_length
pos_embedding_layer = torch.nn.Embedding(context_length, output_dim)

pos_embeddings = pos_embedding_layer(torch.arange(max_length))
print(pos_embeddings.shape)
```

--------------------------------

### Get Final GPU Memory and Training Time Statistics

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Calculates and prints the total training time in seconds and minutes, along with the peak reserved GPU memory usage (both absolute and as a percentage of total memory). It also shows the memory specifically used for LoRA training.

```Python
#@title Show final memory and time stats
used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
used_memory_for_lora = round(used_memory - start_gpu_memory, 3)
used_percentage = round(used_memory         /max_memory*100, 3)
lora_percentage = round(used_memory_for_lora/max_memory*100, 3)
print(f"{trainer_stats.metrics['train_runtime']} seconds used for training.")
print(f"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.")
print(f"Peak reserved memory = {used_memory} GB.")
print(f"Peak reserved memory for training = {used_memory_for_lora} GB.")
print(f"Peak reserved memory % of max memory = {used_percentage} %.")
print(f"Peak reserved memory for training % of max memory = {lora_percentage} %.")

```

--------------------------------

### Generate Text with Nepali Language Model

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

A function to generate text using a trained language model. It takes the model, a starting token, and optional parameters like maximum length and temperature. The function iteratively predicts the next token based on the model's output and converts token IDs back to words.

```Python
# Assuming model, vocab, idx_to_word are defined elsewhere

def generate_text(model, start_token, max_length=50, temperature=1.0):
    model.eval()
    generated = [start_token]
    input_token = torch.tensor([start_token], dtype=torch.long).unsqueeze(0)
    hidden = None

    for _ in range(max_length):
        output, hidden = model(input_token, hidden)
        output = output.squeeze(0).div(temperature).exp()
        next_token = torch.multinomial(output, num_samples=1).item()
        generated.append(next_token)
        input_token = torch.tensor([[next_token]], dtype=torch.long)

        if next_token == vocab.get('<EOS>', -1):
            break

    return ' '.join([idx_to_word.get(idx, "<UNK>") for idx in generated])
```

--------------------------------

### Initialize Tokenizers

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Initializes instances of the WordPiece and SentencePiece tokenizers and specifies the path to the input data file.

```python
# Initialize tokenizers
tokenizer_wp = WordPiece()
tokenizer_sp = SentencePiece()
file_path='chunkeddataset.txt'
```

--------------------------------

### Load Text and Create DataLoader

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Loads text from 'the-verdict.txt', creates a DataLoader using `create_dataloader_v1` with specific parameters, and then retrieves and prints the first batch of data.

```Python
with open("the-verdict.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()

import torch
print("PyTorch version:", torch.__version__)
dataloader = create_dataloader_v1(
    raw_text, batch_size=1, max_length=4, stride=1, shuffle=False
)

data_iter = iter(dataloader)
first_batch = next(data_iter)
print(first_batch)

second_batch = next(data_iter)
print(second_batch)
```

--------------------------------

### Main Execution Pipeline

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Orchestrates the Nepali LLM pipeline, including training a SentencePiece tokenizer, loading a dataset, tokenizing sentences using NepaliTokenizer, and converting to a Hugging Face Dataset format.

```python
# Run the pipeline
from nepali_tokenizer import NepaliTokenizer
if __name__ == "__main__":
    # File paths
    nepali_text_file = "C:/Users/L E G I O N/Desktop/Nepali-LLM/data/merged_nepali_corpus.txt"  # Path to a file with Nepali sentences
    spm_model_prefix = "nepali_spm"

    # Train SentencePiece Tokenizer
    train_sentencepiece_tokenizer(nepali_text_file, spm_model_prefix)

    # Load Dataset
    dataset = load_nepali_dataset()

    # Tokenize Dataset
  
from datasets import Dataset

# Initialize the NepaliTokenizer
tokenizer = NepaliTokenizer()

# Function to load Nepali sentences from a .txt file
def load_nepali_text_file(nepali_text_file):
    """
    Load Nepali text from a .txt file and return a list of sentences.
    """
    with open(nepali_text_file, 'r', encoding='utf-8') as f:
        # Read all lines from the file
        lines = f.readlines()
    
    # Clean up the lines: Remove any unnecessary newlines and empty lines
    sentences = [line.strip() for line in lines if line.strip()]
    return sentences

# Function to tokenize the dataset using NepaliTokenizer
def tokenize_dataset(sentences):
    """
    Tokenizes each Nepali sentence using NepaliTokenizer.
    """
    def tokenize_fn(text):
        # Tokenizing the text using NepaliTokenizer
        return tokenizer.tokenizer(text)
    
    # Tokenizing each sentence
    tokenized_sentences = [tokenize_fn(sentence) for sentence in sentences]
    
    return tokenized_sentences

# Load Nepali text from your local .txt file

sentences = load_nepali_text_file(nepali_text_file)

# Tokenize the sentences using NepaliTokenizer
tokenized_sentences = tokenize_dataset(sentences)

# Convert the tokenized data into a Hugging Face Dataset format
dataset = Dataset.from_dict({"text": sentences, "input_ids": tokenized_sentences})

# Optionally, display the first few entries of the dataset
print(dataset[:5])  # Print first 5 tokenized entries
```

--------------------------------

### Create DataLoader with Different Batch Size and Stride

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Creates a DataLoader with a batch size of 8 and a stride of 4. It then retrieves a batch of input and target sequences and prints them.

```Python
dataloader = create_dataloader_v1(raw_text, batch_size=8, max_length=4, stride=4, shuffle=False)

data_iter = iter(dataloader)
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)
print("\nTargets:\n", targets)
```

--------------------------------

### Import Libraries

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Imports necessary libraries for tokenizer training, model loading, dataset manipulation, and evaluation metrics.

```python
import os
import sentencepiece as spm
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
from datasets import load_dataset, DatasetDict
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
```

--------------------------------

### Import Tokenization Libraries

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Imports the 'pickle' module for data serialization and specific tokenizers ('WordPiece', 'SentencePiece') from the 'nepalitokenizers' library, along with the 'Dataset' class from 'datasets'.

```python
import pickle
from nepalitokenizers import WordPiece, SentencePiece
from datasets import Dataset
```

--------------------------------

### Configure and Initialize UnslothTrainer

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Initializes the UnslothTrainer for continued pretraining. It configures training arguments such as batch size, learning rate, optimizer, and mixed precision (FP16/BF16) based on hardware support. The `embedding_learning_rate` is set to be significantly lower than the main `learning_rate` for effective continual pretraining.

```Python
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported
from unsloth import UnslothTrainer, UnslothTrainingArguments

trainer = UnslothTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    dataset_num_proc = 8,

    args = UnslothTrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 8,

        warmup_ratio = 0.1,
        num_train_epochs = 1,

        learning_rate = 5e-5,
        embedding_learning_rate = 5e-6,

        fp16 = not is_bfloat16_supported(),
        bf16 = is_bfloat16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.00,
        lr_scheduler_type = "cosine",
        seed = 3407,
        output_dir = "outputs",
        report_to = "none", # Use this for WandB etc
    ),
)

```

--------------------------------

### Load Nepali Dataset

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Loads the prepared Nepali training and testing datasets from CSV files ('nepali_train.csv', 'nepali_test.csv') into a DatasetDict object.

```python
# Step 2: Load Dataset
def load_nepali_dataset():
    """
    Load or create a Nepali dataset for the classification task.
    """
    # Replace this with actual data loading logic
    dataset = load_dataset("csv", data_files={"train": "nepali_train.csv", "test": "nepali_test.csv"})
    return DatasetDict({"train": dataset["train"], "test": dataset["test"]})
```

--------------------------------

### Exercise: Encode and Decode Simple String

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Encodes a simple string 'Akwirw ier' into tokens and then decodes it back, demonstrating basic tokenization functionality.

```Python
integers = tokenizer.encode("Akwirw ier")
print(integers)

strings = tokenizer.decode(integers)
print(strings)
```

--------------------------------

### Tokenization and Saving Pipeline

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Demonstrates a pipeline for tokenizing Nepali text using SentencePiece and WordPiece, saving the tokenized data as PKL files, and saving the vocabulary. It also includes loading text and tokenizing datasets.

```python
# Load Nepali text from a .txt file
sentences = load_nepali_text_file(file_path)

# Tokenize the dataset using SentencePiece
input_ids_sp, tokens_sp = tokenize_dataset(sentences, tokenizer_sp)

# Save tokenized data as .pkl
save_as_pkl({
    "input_ids": input_ids_sp,
    "tokens": tokens_sp
}, "sentencepiece_tokenized.pkl")

# Save SentencePiece vocabulary
sp_vocab = tokenizer_sp.get_vocab()  # Assuming get_vocab returns a list of vocabulary tokens
save_vocab(sp_vocab, "sentencepiece_vocab.vocab")

# Tokenize the dataset using WordPiece
input_ids_wp, tokens_wp = tokenize_dataset(sentences, tokenizer_wp)

# Save tokenized data as .pkl
save_as_pkl({
    "input_ids": input_ids_wp,
    "tokens": tokens_wp
}, "wordpiece_tokenized.pkl")

# Save WordPiece vocabulary
wp_vocab = tokenizer_wp.get_vocab()  # Assuming get_vocab returns a list of vocabulary tokens
save_vocab(wp_vocab, "wordpiece_vocab.vocab")

# Optionally, display a confirmation message
print("Tokenized data and vocab files saved successfully.")
```

--------------------------------

### Load and Tokenize Text File

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Reads a text file named 'the-verdict.txt', tokenizes its content using the BPE tokenizer, and prints the total number of tokens.

```Python
with open("the-verdict.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()

enc_text = tokenizer.encode(raw_text)
print(len(enc_text))
```

--------------------------------

### Load and Inspect Pickle Data

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

This Python script demonstrates how to load data from a .pkl file using the pickle library. It includes error handling for different data types (list, dict) and prints information about the loaded content.

```python
import pickle

# Path to your .pkl file
file_path = 'C:/Users/L E G I O N/Desktop/Nepali-LLM/Preprocessing/wordpiece_tokenized.pkl'

# Open and load the file
with open(file_path, 'rb') as f:
    data = pickle.load(f)

# Print the type of the loaded object
print(f"Loaded data type: {type(data)}")

# Inspect the data
if isinstance(data, list):
    print(f"List with {len(data)} elements. First element:")
    print(data[0])
elif isinstance(data, dict):
    print(f"Dictionary with keys: {list(data.keys())}")
    # Example: Inspect the 'train' key if it exists
    if 'input_ids' in data:
        print(f"'train' key contains {len(data['input_ids'])} sequences. First sequence:")
        print(data['input_ids'][0])
else:
    print("Loaded data:")
    print(data)
```

--------------------------------

### Prepare and Split Dataset

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Reads a text file, splits it into training and testing sets based on a specified ratio (20% for testing), and writes the data into CSV files ('nepali_train.csv', 'nepali_test.csv').

```python
import csv
import random

input_file = 'chunkeddataset.txt'
train_file = 'nepali_train.csv'
test_file = 'nepali_test.csv'
test_size_ratio = 0.2  # 20% for testing

# Process the text file and split into train/test datasets
with open(input_file, 'r', encoding='utf-8') as infile, \
     open(train_file, 'w', encoding='utf-8', newline='') as train_out, \
     open(test_file, 'w', encoding='utf-8', newline='') as test_out:
    
    train_writer = csv.writer(train_out)
    test_writer = csv.writer(test_out)
    
    # Write headers
    train_writer.writerow(['text'])
    test_writer.writerow(['text'])
    
    # Read and process line by line
    for line in infile:
        line = line.strip()
        if line:  # Skip empty lines
            if random.random() < test_size_ratio:
                test_writer.writerow([line])
            else:
                train_writer.writerow([line])
```

--------------------------------

### Create Input and Target Sequences

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Creates input (x) and target (y) sequences from a tokenized text sample using a fixed context size. The target sequence is a shifted version of the input sequence.

```Python
enc_sample = enc_text[50:]

context_size = 4

x = enc_sample[:context_size]
y = enc_sample[1:context_size+1]

print(f"x: {x}")
print(f"y:      {y}")
```

--------------------------------

### Run Language Model for Text Generation

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

This snippet demonstrates how to run a language model to generate text, simulating a 'Tiny Stories' style. It utilizes the `transformers` library for tokenization and generation, with output streamed using `TextIteratorStreamer` and printed with text wrapping.

```python
from transformers import TextIteratorStreamer
from threading import Thread
import textwrap

# Assuming tokenizer and model are already loaded and configured
# tokenizer = ...
# model = ...

text_streamer = TextIteratorStreamer(tokenizer)
max_print_width = 100

inputs = tokenizer(
    [
        "Once upon a time, in a galaxy, far far away,"
    ]*1, return_tensors = "pt"
).to("cuda")

generation_kwargs = dict(
    inputs,
    streamer = text_streamer,
    max_new_tokens = 256,
    use_cache = True,
)

thread = Thread(target = model.generate, kwargs = generation_kwargs)
thread.start()

length = 0
for j, new_text in enumerate(text_streamer):
    if j == 0:
        wrapped_text = textwrap.wrap(new_text, width = max_print_width)
        length = len(wrapped_text[-1])
        wrapped_text = "\n".join(wrapped_text)
        print(wrapped_text, end="")
    else:
        length += len(new_text)
        if length >= max_print_width:
            length = 0
            print()
        print(new_text, end="")
    pass
pass
```

--------------------------------

### Apply Tokenizer to Full Text

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Applies the refined tokenization process to the entire 'raw_text' content, cleaning and filtering the tokens, and then prints the first 30 tokens.

```Python
preprocessed = re.split(r'([,.:;?_!"()']|--|\s)', raw_text)
preprocessed = [item.strip() for item in preprocessed if item.strip()]
print(preprocessed[:30])
```

--------------------------------

### Load Tokenizer from Pickle

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/checking.ipynb

Loads a pre-trained tokenizer from a pickle file ('wordpiece_tokenized.pkl'). Includes error handling for the loading process.

```Python
import torch
import pickle
from transformers import PreTrainedTokenizerFast, AutoModelForCausalLM

# === Step 1: Load the Tokenizer (from .pkl) ===
print("Loading tokenizer...")
try:
    with open("wordpiece_tokenized.pkl", "rb") as f:
        tokenizer = pickle.load(f)
    print("Tokenizer loaded successfully!")
except Exception as e:
    print("Error loading tokenizer:", e)
    exit()
```

--------------------------------

### Create DataLoader for GPT Model

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

A function to create a PyTorch DataLoader for the GPTDatasetV1. It takes raw text, batch size, sequence length, and stride as input, and returns a DataLoader configured for training.

```Python
def create_dataloader_v1(txt, batch_size=4, max_length=256, 
                         stride=128, shuffle=True, drop_last=True,
                         num_workers=0):

    # Initialize the tokenizer
    tokenizer = tiktoken.get_encoding("gpt2")

    # Create dataset
    dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)

    # Create dataloader
    dataloader = DataLoader(
        dataset,
        batch_size=batch_size,
        shuffle=shuffle,
        drop_last=drop_last,
        num_workers=num_workers
    )

    return dataloader
```

--------------------------------

### Print First 5 Stories from Tiny Stories Dataset

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Iterates through the first 5 entries in the 'Text' field of the prepared dataset and prints each entry, separated by '==========' to distinguish them.

```Python
for row in dataset[:5]["Text"]:
    print("=========================")
    print(row)

```

--------------------------------

### Print Vocabulary Entries

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Iterates through the created vocabulary and prints its items (token-integer pairs), limiting the output to the first 51 entries.

```Python
for i, item in enumerate(vocab.items()):
    print(item)
    if i >= 50:
        break
```

--------------------------------

### Create Token Embeddings Layer

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Initializes a PyTorch `nn.Embedding` layer to create token embeddings. It defines the vocabulary size and the output dimension for the embeddings.

```Python
input_ids = torch.tensor([2, 3, 5, 1])


vocab_size = 6
output_dim = 3

torch.manual_seed(123)
embedding_layer = torch.nn.Embedding(vocab_size, output_dim)
```

--------------------------------

### Load FastLanguageModel with 4-bit Quantization

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Loads a pre-trained language model using Unsloth's FastLanguageModel, with options for sequence length, data type, and 4-bit quantization for reduced memory usage. Supports various models like Mistral, Llama-3, and Gemma.

```Python
from unsloth import FastLanguageModel
import torch
max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.

# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
fourbit_models = [
    "unsloth/mistral-7b-v0.3-bnb-4bit",      # New Mistral v3 2x faster!
    "unsloth/mistral-7b-instruct-v0.3-bnb-4bit",
    "unsloth/llama-3-8b-bnb-4bit",           # Llama-3 15 trillion tokens model 2x faster!
    "unsloth/llama-3-8b-Instruct-bnb-4bit",
    "unsloth/llama-3-70b-bnb-4bit",
    "unsloth/Phi-3-mini-4k-instruct",        # Phi-3 2x faster!
    "unsloth/Phi-3-medium-4k-instruct",
    "unsloth/mistral-7b-bnb-4bit",
    "unsloth/gemma-7b-bnb-4bit",             # Gemma 2.2x faster!
] # More models at https://huggingface.co/unsloth

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/gemma-7b-bnb-4bit", # "unsloth/mistral-7b" for 16bit loading
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)
```

--------------------------------

### Create Token-to-Integer Mapping

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Creates a dictionary mapping each unique token in the vocabulary to a unique integer ID.

```Python
vocab = {token:integer for integer,token in enumerate(all_words)}
```

--------------------------------

### Read Text File and Print Initial Characters

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Reads a text file named 'dataset.txt' with UTF-8 encoding, prints the total character count, and displays the first 99 characters.

```Python
with open("dataset.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()
    
print("Total number of character:", len(raw_text))
print(raw_text[:99])
```

--------------------------------

### Load and Use Hugging Face Tokenizer

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Loads a pre-trained Nepali tokenizer from Hugging Face ('Sakonii/deberta-base-nepali'), retrieves its maximum sequence length, tokenizes the raw text, and splits the token IDs into chunks based on the maximum length.

```Python
from transformers import AutoTokenizer

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained('Sakonii/deberta-base-nepali')



# Get the maximum sequence length for the model
max_length = tokenizer.model_max_length  # Typically 512 tokens

# Tokenize the text
ids = tokenizer.encode(raw_text)

# Split token IDs into manageable chunks
chunks = [ids[i:i + max_length] for i in range(0, len(ids), max_length)]
```

--------------------------------

### Train Nepali Language Model

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Implements a training loop for a Nepali language model. It iterates through epochs and batches, calculates loss using CrossEntropyLoss, performs forward and backward passes, and updates the model weights. Includes a progress bar for monitoring batch progress within each epoch and displays the loss for each epoch.

```Python
from tqdm import tqdm
import torch

# Assuming model, optimizer, criterion, vocab_size, input_ids, batch_size, num_epochs, get_batches are defined elsewhere

for epoch in range(num_epochs):
    total_loss = 0
    hidden = None

    batch_progress = tqdm(get_batches(input_ids, batch_size), total=len(input_ids) // batch_size, desc=f"Epoch {epoch+1}")

    for batch in batch_progress:
        inputs = batch[:, :-1]
        targets = batch[:, 1:]

        optimizer.zero_grad()
        
        outputs, hidden = model(inputs, hidden)

        outputs_reshaped = outputs.contiguous().view(-1, vocab_size)
        targets_reshaped = targets.contiguous().view(-1)

        assert outputs_reshaped.size(0) == targets_reshaped.size(0), "Shape mismatch between outputs and targets."

        loss = criterion(outputs_reshaped, targets_reshaped)
        
        loss.backward()
        optimizer.step()

        hidden = None

        total_loss += loss.item()

        batch_progress.set_postfix(loss=loss.item())

    print(f"Epoch {epoch+1}/{num_epochs}, Loss: {total_loss:.4f}")
```

--------------------------------

### Mount Google Drive

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Mounts the Google Drive to the Colab environment, allowing access to files stored in Google Drive.

```Python
from google.colab import drive
drive.mount('/content/drive')
```

--------------------------------

### SimpleTokenizerV2 Implementation

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

A Python class for tokenizing text. It handles unknown words by replacing them with '<|unk|>' and also preprocesses text by replacing spaces before specific punctuation marks. It includes methods for encoding text into token IDs and decoding token IDs back into text.

```python
import re

class SimpleTokenizerV2:
    def __init__(self, vocab):
        self.str_to_int = vocab
        self.int_to_str = { i:s for s,i in vocab.items()}
    
    def encode(self, text):
        preprocessed = re.split(r'([,.:;?_!"()'']|--|s)', text)
        preprocessed = [item.strip() for item in preprocessed if item.strip()]
        preprocessed = [
            item if item in self.str_to_int 
            else "<|unk|>" for item in preprocessed
        ]

        ids = [self.str_to_int[s] for s in preprocessed]
        return ids
        
    def decode(self, ids):
        text = " ".join([self.int_to_str[i] for i in ids])
        # Replace spaces before the specified punctuations
        text = re.sub(r'\s+([,.:;?"!()''])', r'\1', text)
        return text
```

--------------------------------

### Initialize Positional Embeddings Layer

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Initializes a PyTorch `nn.Embedding` layer to create positional embeddings. It defines the maximum context length and the output dimension for the embeddings.

```Python
vocab_size = 50257
output_dim = 256

token_embedding_layer = torch.nn.Embedding(vocab_size, output_dim)
```

--------------------------------

### Print Last 5 Vocabulary Entries

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Prints the last five key-value pairs from the updated vocabulary to verify the addition of the special tokens and check the new vocabulary size.

```python
for i, item in enumerate(list(vocab.items())[-5:]):
    print(item)
```

--------------------------------

### Import TensorFlow Libraries

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Imports necessary libraries from TensorFlow for building and training neural network models, including layers, optimizers, and preprocessing utilities.

```python
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical
```

--------------------------------

### Train Transformer Model

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

This snippet shows a placeholder for training a Transformer model using a dataset and tokenizer. It's a high-level representation of the training initiation.

```python
trained_model = train_transformer_model(dataset, tokenizer)
```

--------------------------------

### Apply PEFT LoRA Adapters

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/colabcode.ipynb

Applies Parameter-Efficient Fine-Tuning (PEFT) LoRA adapters to the loaded model. This significantly reduces the number of trainable parameters and memory usage. It includes options for rank, target modules, LoRA alpha, dropout, and gradient checkpointing.

```Python
model = FastLanguageModel.get_peft_model(
    model,
    r = 128, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",

                      "embed_tokens", "lm_head",], # Add for continual pretraining
    lora_alpha = 32,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 3407,
    use_rslora = True,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)
```

--------------------------------

### Create Vocabulary from Unique Tokens

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Generates a sorted list of unique tokens from the preprocessed text and calculates the vocabulary size.

```Python
all_words = sorted(set(preprocessed))
vocab_size = len(all_words)

print(vocab_size)
```

--------------------------------

### Generate Context-Target Pairs

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Iterates through a tokenized text sample to generate context and target pairs. For each position, it creates a context (sequence up to that point) and the next token as the target.

```Python
for i in range(1, context_size+1):
    context = enc_sample[:i]
    desired = enc_sample[i]

    print(context, "---->", desired)

for i in range(1, context_size+1):
    context = enc_sample[:i]
    desired = enc_sample[i]

    print(tokenizer.decode(context), "---->", tokenizer.decode([desired]))
```

--------------------------------

### Load Model and Generate Text (Alternative Tokenizer Path)

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/checking.ipynb

Loads a pre-trained causal language model and generates text. This snippet demonstrates loading the tokenizer from a specific local path and also from the root directory. It includes error handling for model loading and text generation.

```Python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("C:/Users/L E G I O N/Desktop/Nepali-LLM/Preprocessing/")

print("Loading model...")
try:
    # Map to CPU
    model = torch.load("model.pt", map_location=torch.device('cpu'))
    model.eval()  # Set to evaluation mode
    print("Model loaded successfully!")
except Exception as e:
    print("Error loading model:", e)
    exit()

# === Step 3: Prepare Input Text ===
input_text = "नेपालको राजधानी काठमाडौँ हो।"
print(f"Input text: {input_text}")

# Load the tokenizer corresponding to the model
tokenizer = AutoTokenizer.from_pretrained("/")

# Tokenize the input
try:
    input_ids = tokenizer.encode(input_text, return_tensors="pt")
    print("Input tokenized successfully!")
except Exception as e:
    print("Error tokenizing input:", e)
    exit()

# === Step 4: Generate Text ===
print("Generating text...")
try:
    with torch.no_grad():
        outputs = model.generate(
            input_ids=input_ids,
            max_length=50,  # Maximum length of generated text
            num_beams=5,  # Beam search with 5 beams
            no_repeat_ngram_size=2,  # Avoid repeating n-grams
            early_stopping=True
        )
    # Decode and print the output
    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print("Generated Text:", generated_text)
except Exception as e:
    print("Error during text generation:", e)
    exit()

# === Step 5: Optional Evaluation ===
# If you have ground truth data, you can calculate BLEU or ROUGE scores here.
# Example:
# reference = ["नेपालको राजधानी काठमाडौँ हो।"]
# print("Evaluation metrics (optional) can be added here.")
```

--------------------------------

### Display Tokenized Data

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Prints the first 5 tokenized IDs and tokens for both SentencePiece and WordPiece processed data.

```python
print("SentencePiece Tokenized IDs:", input_ids_sp[:5])
print("SentencePiece Tokens:", tokens_sp[:5])
print("WordPiece Tokenized IDs:", input_ids_wp[:5])
print("WordPiece Tokens:", tokens_wp[:5])
```

--------------------------------

### SimpleTokenizerV1 Class

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

A custom tokenizer class that handles encoding text into token IDs and decoding token IDs back into text. It uses a pre-defined vocabulary for mapping.

```Python
class SimpleTokenizerV1:
    def __init__(self, vocab):
        self.str_to_int = vocab
        self.int_to_str = {i:s for s,i in vocab.items()}
    
    def encode(self, text):
        preprocessed = re.split(r'([,.:;?_!"()']|--|\s)', text)
                                
        preprocessed = [
            item.strip() for item in preprocessed if item.strip()
        ]
        ids = [self.str_to_int[s] for s in preprocessed]
        return ids
        
    def decode(self, ids):
            text = " ".join([self.int_to_str[i] for i in ids])
            # Replace spaces before the specified punctuations
            text = re.sub(r'\s+([,.?!"()'])', r'\1', text)
            return text
```

--------------------------------

### GemmaCausalLM Text Generation with Timing

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/training/gemma.ipynb

This snippet shows how to load the GemmaCausalLM model from keras_nlp, set up timing functions (tick and tock), and implement a text generation function that utilizes the model and timing.

```python
import keras
import keras_nlp

import time

gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(model_id)
gemma_lm.summary()

tick_start = 0

def tick():
    global tick_start
    tick_start = time.time()

def tock():
    print(f"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s")

def text_gen(prompt):
    tick()
    input = f"user\n{prompt}\nmodel\n"
    output = gemma_lm.generate(input, max_length=token_limit)
    print("\nGemma output:")
    print(output)
    tock()
```

--------------------------------

### Print Tokenized Chunks

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Archive/Preprocessing-Tokenizer/Tokenizer.ipynb

Iterates through a list of tokenized text chunks and prints each chunk with its index.

```python
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}/{len(chunks)}: {chunk}")
```

--------------------------------

### Train SentencePiece Tokenizer

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Trains a SentencePiece tokenizer using the Byte Pair Encoding (BPE) algorithm for the Nepali language. It saves the trained model and vocabulary.

```python
# Step 1: Train SentencePiece Tokenizer
def train_sentencepiece_tokenizer(data_file, model_prefix, vocab_size=32000):
    """
    Train a SentencePiece tokenizer for the Nepali language.
    """
    spm.SentencePieceTrainer.train(
        input=data_file,
        model_prefix=model_prefix,
        vocab_size=vocab_size,
        model_type="bpe",  # Can be "unigram", "bpe", etc.
        character_coverage=0.9995,  # Adjust for Devanagari
        pad_id=0,
        unk_id=1,
        bos_id=2,
        eos_id=3
    )
    print(f"SentencePiece model trained and saved as {model_prefix}.model")
```

--------------------------------

### Load Nepali Text File

Source: https://github.com/1shanpanta/nepali-llm/blob/main/Dataset Tasks/Preprocessing/prep.ipynb

Reads Nepali text from a specified .txt file, cleans each line by removing leading/trailing whitespace and empty lines, and returns a list of sentences.

```python
# Function to load Nepali text from a .txt file
def load_nepali_text_file(file_path):
    """
    Load Nepali text from a .txt file and return a list of sentences.
    """
    with open(file_path, 'r', encoding='utf-8') as f:
        # Read all lines from the file
        lines = f.readlines()
    
    # Clean up the lines: Remove any unnecessary newlines and empty lines
    sentences = [line.strip() for line in lines if line.strip()]
    return sentences
```