### Install Project Dependencies

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Installs all required packages listed in 'requirement.txt' and the 'datasets' library. This step is crucial for running the project's scripts.

```python
%%capture
!pip install -r requirement.txt
!pip install datasets 
```

--------------------------------

### Install Dependencies

Source: https://github.com/arbml/qawafi/blob/main/diac_inference.ipynb

Installs the required Python packages from the requirements file while suppressing output.

```python
%%capture
!pip install -r requirement.txt
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Installs essential Python packages for the project, including 'datasets', 'arabic-reshaper', 'python-bidi', and custom libraries 'tnkeeh' and 'tkseem' from GitHub.

```python
%%capture
!pip install datasets
!pip install --upgrade arabic-reshaper
!pip install python-bidi
!pip install git+https://github.com/ARBML/tnkeeh.git
!pip install git+https://github.com/ARBML/tkseem.git
```

--------------------------------

### Install Python Packages

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Installs essential Python libraries for natural language processing and deep learning, including sentence-transformers, datasets, and tnkeeh.

```python
!pip install -U sentence-transformers
!pip install datasets 
!pip install tnkeeh
!pip install git+https://github.com/ARBML/tnkeeh.git
```

--------------------------------

### Bait Analysis Output Format

Source: https://github.com/arbml/qawafi/blob/main/README.md

Example structure of the output returned by the BaitAnalysis module.

```json
{'arudi_style'     : [['ألاليت شعري هل أبيتنن ليلتن بجنب لغضى أزجلقلاص ننواجيا',
                     '1101011010101101011011011010110101011010110110']],
 'closest_baits'   : [[('ألاليت شعري هل أبيتن ليلة # بجنب الغضى أزجي القِلاص '
                     'النواجيا',
                     [0.38896721601486206])]],
 'closest_patterns': [('1101011010101101011011011010110101011010110110',
                       1.0,
                       'فعولنْ مفاعيلنْ فعولنْ مفاعلنْ # فعولنْ مفاعيلنْ '
                       'فعولنْ مفاعلنْ')],
 'diacritized'     : ['أَلَالَيْتُ شِعْرِي هَلْ أَبِيتَنَّ لَيْلَةً # بِجَنْبِ '
                     'الْغَضَى أَزْجِي الْقِلَاصَ النَّوَاجِيَا'],
 'era'             : ['العصر الحديث', 'العصر العثماني'],
 'meter'           : 'الطويل',
 'qafiyah'         : ('ي',
                     'قافية بحرف الروي: ي ،  زاد لها الوصل بإشباع رويها زاد لها '
                     'التأسيس'),
 'theme'           : ['قصيدة رومنسيه', 'قصيدة شوق', 'قصيدة غزل']}
```

--------------------------------

### Model Checkpointing Setup

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Configures model checkpointing to save the model's weights during training. It monitors validation loss and saves the best performing weights.

```python
checkpoint_path = "/content/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

callbacks = [tf.keras.callbacks.ModelCheckpoint(checkpoint_path,
                                                 monitor='val_loss',
                                                 verbose=1,
                                                 save_weights_only=True,
                                                 save_best_only= True,
                                                 mode='min')]
```

--------------------------------

### Load 'ashaar' Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Loads the 'ashaar' dataset from Hugging Face using the 'datasets' library. This dataset contains Arabic poetry. Ensure 'datasets' is installed before running.

```python
import pandas as pd
import json
import datasets
ashaar = datasets.load_dataset('MagedSaeed/ashaar')
```

--------------------------------

### Create Character Index Mapping

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Generates a dictionary mapping characters to unique indices starting from 1.

```python
char2idx = {u:i+1 for i, u in enumerate(vocab)}
```

--------------------------------

### Clone Arabic Diacritization Repository

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Clones the official Arabic Diacritization GitHub repository and changes the current directory to the cloned repository. Ensure you have Git installed.

```python
!git clone https://github.com/zaidalyafeai/Arabic_Diacritization
%cd Arabic_Diacritization
```

--------------------------------

### Initialize and Summarize Model

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Create the model architecture and display its layer summary.

```python
model = create_model()
model.summary()
```

--------------------------------

### Initialize Model

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/theme.ipynb

Instantiates the model using the defined architecture.

```python
model = create_model()
```

--------------------------------

### Create Directory

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Creating directories for data storage using shell commands.

```bash
!mkdir -p data/CA_MSA
```

```bash
!mkdir -p ashaar/CA_MSA
```

--------------------------------

### Remove Stop Words

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Filters out words present in the global stop_words list from the text field of an example.

```python
def remove_stop_words(example):
    example['text'] = ' '.join([word for word in example['text'].split(' ') if word not in stop_words])
      
    return example
```

--------------------------------

### Import Libraries and Load Poetry Samples

Source: https://github.com/arbml/qawafi/blob/main/qawafi_server/tests.ipynb

Imports necessary libraries and loads a large collection of Arabic poetry samples. Sets a random seed for reproducibility.

```python
import json
import random
import sys
import time
from pprint import pprint

import requests

sys.path.append("..\\")
from bohour.poem_samples_large import samples


random.seed(1)

qasaed = [random.choice(samples) for _ in range(10)]
```

--------------------------------

### Get Label Value Counts

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Calculates and returns the frequency of each unique label in the DataFrame. Useful for understanding data imbalance.

```python
df['label'].value_counts()
```

--------------------------------

### Perform Complete Poetry Analysis with BaitAnalysis

Source: https://context7.com/arbml/qawafi/llms.txt

Initializes the BaitAnalysis engine to process verses and extract prosodic features. Requires the qawafi_server package and loads deep learning models upon instantiation.

```python
from qawafi_server.bait_analysis import BaitAnalysis

# Initialize the analyzer (loads all deep learning models)
analysis = BaitAnalysis(use_cbhg=True)

# Prepare input baits - each bait has two shatrs (hemistichs) separated by #
baits = [
    "القلب أعلم يا عذول بدائه # وأحق منك بجفنه وبمائه",
    "لا تعذل المشتاق في أشواقه # حتى يكون حشاك في أحشائه"
]

# Perform complete analysis
output = analysis.analyze(
    baits=baits,
    short_qafiyah=False,        # Use detailed qafiyah analysis
    override_tashkeel=True,     # Override auto-diacritization with user input
    highlight_output=False,     # Disable colored terminal output
    predict_era=True,           # Enable era classification
    predict_theme=True,         # Enable theme classification
    predict_closest=True        # Find similar baits from corpus
)
```

--------------------------------

### Preview JSON Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Displays the first 10 lines of the generated JSON dataset file.

```bash
!head -10 /content/era_dataset.json
```

--------------------------------

### Load Model Weights

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Instantiates the model using create_model() and loads pre-trained weights from a checkpoint file.

```python
model = model = create_model()
model.load_weights('model/cp.ckpt')
```

--------------------------------

### Initialize Tester

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Setting up the DiacritizationTester with model configuration.

```python
from predict import DiacritizationTester
tester = DiacritizationTester('config/cbhg2.yml', 'cbhg')
```

--------------------------------

### Download Pretrained Models

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Downloads the meters dataset from Google Drive using gdown.

```python
import gdown 
print("Exporting the pretrained models ... ")
url = 'https://drive.google.com/uc?id=11iIHChBR7sVcUfGMnxfEAjbe7sSjzx5M' 
gdown.cached_download(url,'meters_dataset.zip', quiet=False, postprocess=gdown.extractall)
```

--------------------------------

### Load Qawafi Dataset

Source: https://context7.com/arbml/qawafi/llms.txt

Loads the Zaid/metrecv2 dataset with a specific subset configuration.

```python
dataset = load_dataset("Zaid/metrecv2", "train_50k")
```

--------------------------------

### Initialize Data Generators

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Instantiates training and validation generators with appropriate augmentation settings.

```python
train_generator = DataGenerator(X_train, y_train)
valid_generator = DataGenerator(X_valid, y_valid, augment = False, shuffle=False)
```

--------------------------------

### Preprocess and Tokenize Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Prepares text data by extracting samples, training a SentencePiece tokenizer, and mapping input IDs to the dataset.

```python
# dataset = dataset.map(remove_stop_words)
examples = copy.deepcopy(dataset)


data = []
for sample in dataset['train']:
    data.append(sample['text'])

open(f'/content/data.txt', 'w').write(('\n').join(data))
tokenizer = tk.SentencePieceTokenizer
tokenizer = tokenizer(vocab_size = vocab_size)
tokenizer.train('/content/data.txt')
dataset = dataset.map(lambda examples:{'input_ids': tokenizer.encode_sentences(examples['text'], out_length= max_tokens)}, batched=True)
columns=['input_ids', 'labels'] 
dataset = dataset.map(lambda examples:{'labels': examples['label']}, batched=True)
```

--------------------------------

### Train the Model

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Execute the training process using the prepared training and validation datasets.

```python
model.fit(x_train, y_train, validation_data = (x_valid, y_valid) ,epochs = 5, shuffle = True, batch_size=128)
```

--------------------------------

### Load MetRecV2 Dataset

Source: https://github.com/arbml/qawafi/blob/main/README.md

Import the MetRecV2 dataset from Hugging Face using the datasets library.

```python
from datasets import load_dataset
dataset = load_dataset("Zaid/metrecv2", "train_all")
# or the smaller one 
dataset = load_dataset("Zaid/metrecv2", "train_50k")
```

--------------------------------

### Configure Text Preprocessing

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Defines configuration parameters for text cleaning and normalization using the tnkeeh library.

```python
import tnkeeh as tn 
from datasets import load_dataset
import os 
from transformers import AutoTokenizer
import torch
import copy 
import tkseem as tk

data_args = {
    'segment' : False,
    'remove_special_chars' : False,
    'remove_english' : True
,
    'normalize' : False,
    'remove_diacritics' : True
,
    'excluded_chars' : []
  ,  'remove_tatweel' : True
,
    'remove_html_elements' : False,
    'remove_links' : False, 
    'remove_twitter_meta' : False,
    'remove_long_words' : False,
    'remove_repeated_chars' : False,
}
```

--------------------------------

### Load and Prepare Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Loads a JSON dataset and applies cleaning and splitting routines.

```python
dataset = load_dataset('json',data_files=['/content/era_dataset.json'],field='data')
dataset = clean_dataset(dataset)

dataset = split_dataset(dataset)
```

--------------------------------

### Import TensorFlow and Utilities

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Imports core TensorFlow modules, NumPy, and other necessary utilities for building and training deep learning models.

```python
import tensorflow as tf
import numpy as np
import os
import time
import glob
from random import shuffle
from tensorflow.keras.layers import GRU, Embedding, Dense, Input, Dropout, Bidirectional, BatchNormalization, Flatten, Reshape
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import Tokenizer, text_to_word_sequence
from tensorflow.keras.preprocessing.sequence import pad_sequences
from sklearn.utils import shuffle
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
```

--------------------------------

### Analyze tafeelat with Fawlon

Source: https://github.com/arbml/qawafi/blob/main/README.md

Access arudi rules and patterns for the Fawlon tafeela.

```python
# to avoid the pain of paths, add qawafi_server to your python path
# in the current working directory of qawafi, do:
import sys
sys.path.append('qawafi_server/bohour')
# bohour should be in your  path now.
from qawafi_server.bohour.tafeela import Fawlon

fawlon = Fawlon()
fawlon.allowed_zehafs
# >> [bohour.zehaf.Qabadh, bohour.zehaf.Thalm, bohour.zehaf.Tharm]
fawlon.fawlon.all_zehaf_tafeela_forms()
# >> [فعولنْ, فعول, عولنْ, عول]
# for the 0/1 pattern
fawlon.pattern_int
# >> 11010
# to be arabic friendly :)
fawlon.name
# >> 'فعولنْ'
```

--------------------------------

### Load ashaar dataset

Source: https://github.com/arbml/qawafi/blob/main/README.md

Download the ashaar dataset from the Huggingface Datasets Hub.

```python
# pip install datasets
import datasets
ashaar = datasets.load_dataset('arbml/ashaar')
ashaar
```

--------------------------------

### Load Model and Evaluate

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Loads the trained model weights from a checkpoint and evaluates the model's performance on the test dataset. This provides metrics like loss and accuracy.

```python
model = create_transformer_model()
model.load_weights(checkpoint_path)
model.evaluate(X_test, y_test, batch_size = 256)
```

--------------------------------

### Model Creation and Compilation

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Instantiates and compiles the Transformer model using the `create_transformer_model` function. This prepares the model for training.

```python
model = create_transformer_model()
```

--------------------------------

### Prepare Training Data

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Converts the dataset's input IDs into a NumPy array for model training.

```python
x_train = np.array(dataset['train']['input_ids'])
```

--------------------------------

### Train Model

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Executing the training script with specific model and configuration parameters.

```bash
!python train.py --model cbhg --config config/cbhg.yml
```

```bash
!python train.py --model cbhg --config config/cbhg2.yml
```

--------------------------------

### Prepare Training and Validation Arrays

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/theme.ipynb

Converts dataset features into NumPy arrays for model training.

```python
x_train = np.array(dataset['train']['input_ids'])
y_train = np.array(dataset['train']['labels'])
x_valid = np.array(dataset['valid']['input_ids'])
y_valid = np.array(dataset['valid']['labels'])
```

--------------------------------

### Inspect Exported JSON

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/theme.ipynb

Displays the first 10 lines of the exported JSON file.

```bash
!head -10 /content/theme_dataset.json
```

--------------------------------

### Balance Dataset Samples

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Creates a dictionary to define the number of samples per label, capping counts at 5000. Then, it resamples the DataFrame to ensure a balanced distribution based on these counts.

```python
sample_amounts = {val:5000 if cnt>5000 else cnt for val, cnt in df['label'].value_counts().iteritems()}
print(sample_amounts)
df = (
    df.groupby('label').apply(lambda g: g.sample(
        # lookup number of samples to take
        n=sample_amounts[g.name],
        # enable replacement if len is less than number of samples expected
        replace=False  
    ))
)
```

--------------------------------

### Initialize Tokenizer Factory

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Returns a specific tokenizer class based on the provided name string.

```python
def get_tokenizer(tok_name):
    tokenizers = {'SentencePieceTokenizer':tk.SentencePieceTokenizer, 'WordTokenizer':tk.WordTokenizer, 'CharacterTokenizer':tk.CharacterTokenizer,
         'MorphologicalTokenizer':tk.MorphologicalTokenizer, 'RandomTokenizer':tk.RandomTokenizer, 'DisjointLetterTokenizer':tk.DisjointLetterTokenizer}
    return tokenizers[tok_name]
```

--------------------------------

### Load Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Loading datasets from Hugging Face or local JSON files.

```python
import pandas as pd
import datasets
ashaar = datasets.load_dataset('MagedSaeed/ashaar')
```

```python
from datasets import load_dataset
dataset = load_dataset('json',data_files=['/content/dataset.json'],field='data')
```

--------------------------------

### Analyze Kamel meter

Source: https://github.com/arbml/qawafi/blob/main/README.md

Explore sub-meters, arud/dharb mappings, and tafeelat combinations for the Kamel meter.

```python
# continuing from the previous session

from bohour.bahr import Kamel

kamel = Kamel()

# this gives all the possible related bahrs to this bahr, like majzoo and mashtoor, etc.
kamel.sub_bahrs
# >> (bohour.bahr.KamelMajzoo,)

# this gives all the special cases related to this bahr at the end of each of each shatr
# this has been implemented as a mapping as follows
kamel.arod_dharbs_map
"""
>>>
{
 bohour.zehaf.NoZehafNorEllah: (
  bohour.zehaf.NoZehafNorEllah,
  bohour.zehaf.Edmaar,
  bohour.zehaf.Qataa,
  bohour.zehaf.QataaAndEdmaar,
  bohour.zehaf.HathathAndEdmaar
  ),
 bohour.zehaf.Edmaar: (
  bohour.zehaf.NoZehafNorEllah,
  bohour.zehaf.Edmaar,
  bohour.zehaf.Qataa,
  bohour.zehaf.QataaAndEdmaar,
  bohour.zehaf.HathathAndEdmaar
 ),
 bohour.zehaf.Hathath: (bohour.zehaf.Hathath, bohour.zehaf.HathathAndEdmaar)
}
"""
# to show all the possible combinations of tafeelat of this bahr:
kamel.all_combinations
# >>
"""
((متفاعلنْ, متفاعلنْ, متفاعلنْ), (متفاعلنْ, متفاعلنْ, متفاعلنْ)),
 ((متفاعلنْ, متفاعلنْ, متفاعلنْ), (متفاعلنْ, متفاعلنْ, متْفاعلنْ)),
 ((متفاعلنْ, متفاعلنْ, متفاعلنْ), (متفاعلنْ, متفاعلنْ, متفاعلْ)),
 ((متفاعلنْ, متفاعلنْ, متفاعلنْ), (متفاعلنْ, متفاعلنْ, متْفاعلْ)),
 ((متفاعلنْ, متفاعلنْ, متفاعلنْ), (متفاعلنْ, متفاعلنْ, متْفا)),
... etc
"""
# to show the 0/1 combinations of the previous tafeelat:
kamel.all_combinations_patterns
# >> # 
"""
'111011011101101110110111011011101101110110',
 '111011011101101110110111011011101101010110',
 '11101101110110111011011101101110110111010',
 '11101101110110111011011101101110110101010',
 '111011011101101110110111011011101101010',
 '111011011101101110110111011010101101110110',
 '111011011101101110110111011010101101010110',
 '11101101110110111011011101101010110111010',
... etc
"""
```

--------------------------------

### Integrate iOS Client API

Source: https://context7.com/arbml/qawafi/llms.txt

Performs a multipart form-data POST request to the Qawafi server and defines the response model.

```swift
import Foundation

// API call to analyze Arabic poetry
class API {
    static func getAnalysis(_ text: String) async throws -> ResponseNew? {
        let parameters = [["key": "baits", "value": text, "type": "text"]]
        let boundary = "Boundary-\(UUID().uuidString)"

        var body = ""
        for param in parameters {
            body += "--\(boundary)\r\n"
            body += "Content-Disposition:form-data; name=\"\(param["key"]!)\""
            body += "\r\n\r\n\(param["value"]!)\r\n"
        }
        body += "--\(boundary)--\r\n"

        var request = URLRequest(url: URL(string: "https://your-server/api/analyze")!)
        request.addValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")
        request.httpMethod = "POST"
        request.httpBody = body.data(using: .utf8)

        let (data, _) = try await URLSession.shared.data(for: request)
        return try? JSONDecoder().decode(ResponseNew.self, from: data)
    }
}

// Response model
class ResponseNew: Codable {
    let diacritized: [String]
    let arudi_style: [[String]]
    let qafiyah: [String]
    let meter: String
    let era: [String]
    let closest_patterns: [[ClosestPattern]]
    let theme: [String]
}

// Usage
let response = try await API.getAnalysis("القلب أعلم يا عذول بدائه\nوأحق منك بجفنه وبمائه")
print(response?.meter)  // "الكامل"
```

--------------------------------

### Prepare Test Data

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Convert test dataset entries into NumPy arrays.

```python
x_test = np.array(dataset['test']['input_ids'])
y_test = np.array(dataset['test']['labels'])
```

--------------------------------

### Load Weights and Evaluate Model

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Load pre-trained weights and evaluate model performance on the test set.

```python
model = create_model()
model.load_weights(checkpoint_path)
model.evaluate(x_test, y_test, batch_size = 256)
```

--------------------------------

### Download and Extract Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Downloads the meters_dataset.zip archive from Google Drive and extracts its contents.

```python
import gdown 
url = 'https://drive.google.com/uc?id=11iIHChBR7sVcUfGMnxfEAjbe7sSjzx5M' 
gdown.cached_download(url,'meters_dataset.zip', quiet=False, postprocess=gdown.extractall)
```

--------------------------------

### Import Dependencies

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Imports necessary libraries for deep learning, data manipulation, and visualization.

```python
import tensorflow as tf
import numpy as np
import os
import time
import glob
from tensorflow.keras.layers import GRU, Embedding, Dense, Input, Dropout, Bidirectional, BatchNormalization, Flatten, Reshape
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import Tokenizer, text_to_word_sequence
from tensorflow.keras.preprocessing.sequence import pad_sequences
from sklearn.utils import shuffle
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
```

--------------------------------

### Analyze Tafeela (Metrical Foot) Variations

Source: https://context7.com/arbml/qawafi/llms.txt

Utilize the bohour.tafeela module to analyze metrical feet (tafeelat), including their allowed variations (zehaf) and forms.

```python
import sys
sys.path.append('qawafi_server/bohour')

from bohour.tafeela import Fawlon

# Create a tafeela instance
fawlon = Fawlon()

# Get allowed zehaf (metrical variations) for this tafeela
print(fawlon.allowed_zehafs)
# [bohour.zehaf.Qabadh, bohour.zehaf.Thalm, bohour.zehaf.Tharm]

# Get all possible zehaf forms
print(fawlon.all_zehaf_tafeela_forms())
# [فعولنْ, فعول, عولنْ, عول]

# Get binary pattern representation
print(fawlon.pattern_int)
# 11010

# Get Arabic name
print(fawlon.name)
# 'فعولنْ'
```

--------------------------------

### Create Test Data Function

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

A utility function to prepare test data. It tokenizes text, pads sequences to a fixed length, and converts the data into NumPy arrays.

```python
def create_test_data(X, y):
  X_test = []
  y_test = []

  for text in X:
    x = [[char2idx[char] for char in text]]
    x = pad_sequences(x, padding='post', value=0, maxlen = 128)
    X_test.append(x[0])
  
  return np.array(X_test), np.array(y)
```

--------------------------------

### Analyze Bahr (Meter) Patterns

Source: https://context7.com/arbml/qawafi/llms.txt

Explore complete Arabic meter structures using the bohour.bahr module, including sub-meters, end-of-hemistich variations, and all valid tafeelat combinations.

```python
from bohour.bahr import Kamel

# Create a bahr (meter) instance
kamel = Kamel()

# Get related sub-meters (majzoo, mashtoor, etc.)
print(kamel.sub_bahrs)
# (bohour.bahr.KamelMajzoo,)

# Get arod-dharb mapping (end-of-hemistich variations)
print(kamel.arod_dharbs_map)
# {
#   bohour.zehaf.NoZehafNorEllah: (
#     bohour.zehaf.NoZehafNorEllah,
#     bohour.zehaf.Edmaar,
#     bohour.zehaf.Qataa,
#     ...
#   ),
#   ...
# }

# Get all valid tafeelat combinations for this meter
print(kamel.all_combinations[:3])
# [
#   ((متفاعلنْ, متفاعلنْ, متفاعلنْ), (متفاعلنْ, متفاعلنْ, متفاعلنْ)),
#   ((متفاعلنْ, متفاعلنْ, متفاعلنْ), (متفاعلنْ, متفاعلنْ, متْفاعلنْ)),
#   ...
# ]

# Get binary pattern representations
print(kamel.all_combinations_patterns[:3])
# [
#   '111011011101101110110111011011101101110110',
#   '111011011101101110110111011011101101010110',
#   ...
# ]
```

--------------------------------

### Prepare Training and Validation Data

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Convert dataset dictionary entries into NumPy arrays for model input.

```python
y_train = np.array(dataset['train']['labels'])
x_valid = np.array(dataset['valid']['input_ids'])
y_valid = np.array(dataset['valid']['labels'])
```

--------------------------------

### Balance Dataset Classes

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/theme.ipynb

Calculates sample amounts per class and performs undersampling to balance the dataset.

```python
# min_class = has_accepted_theme_five_baits_df['label'].value_counts().values.min()
# min_class = 500
sample_amounts = {idx:5_000 if cnt > 5_000 else cnt for idx, cnt in has_accepted_theme_five_baits_df['label'].value_counts().iteritems()}
# sample_amounts[2] = min_class
# sample_amounts = {i:(min_class) for i in range(len(labels2theme)) }
print(sample_amounts)
has_accepted_theme_five_baits_df = (
    has_accepted_theme_five_baits_df.groupby('label').apply(lambda g: g.sample(
        # lookup number of samples to take
        n=sample_amounts[g.name],
        # enable replacement if len is less than number of samples expected
        replace=False  
    ))
)
```

--------------------------------

### Load and Clean Hugging Face Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/theme.ipynb

Loads a JSON dataset from specified files and cleans the 'text' field using a Tnkeeh cleaner. Ensure 'data_args' is defined before use.

```python
stop_words = open('/content/list.txt', 'r').read().splitlines() + more_stop_words

def get_tokenizer(tok_name):
    tokenizers = {'SentencePieceTokenizer':tk.SentencePieceTokenizer, 'WordTokenizer':tk.WordTokenizer, 'CharacterTokenizer': tk.CharacterTokenizer,
         'MorphologicalTokenizer':tk.MorphologicalTokenizer, 'RandomTokenizer': tk.RandomTokenizer, 'DisjointLetterTokenizer': tk.DisjointLetterTokenizer}
    return tokenizers[tok_name]


def split_dataset(dataset, seed = 42):
    #create validation split
    if 'valid' not in dataset:
        train_valid_dataset = dataset['train'].train_test_split(test_size=0.1, seed = seed)
        dataset['valid'] = train_valid_dataset.pop('test')
        dataset['train'] = train_valid_dataset['train']

    #create training split 
    if 'test' not in dataset:
        train_valid_dataset = dataset['train'].train_test_split(test_size=0.1, seed = seed)
        dataset['test'] = train_valid_dataset.pop('test')
        dataset['train'] = train_valid_dataset['train']  
    return dataset 


def clean_dataset(dataset):
    
    cleaner = tn.Tnkeeh(**data_args)
    dataset = cleaner.clean_hf_dataset(dataset, 'text')
    return dataset 

def remove_stop_words(example):
    example['text'] = ' '.join([word for word in example['text'].split(' ') if word not in stop_words])
      
    return example

tokenizer_name = ''
max_tokens = 128
vocab_size  = 10_000

batch_size = 512

# clean and load data
dataset = load_dataset('json',data_files=['/content/theme_dataset.json'],field='data')
dataset = clean_dataset(dataset)

dataset = split_dataset(dataset)
```

--------------------------------

### Inspect Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Displaying dataset information and contents.

```python
dataset
```

```python
filtered
```

```python
split_filtered
```

```python
filtered["train"]["poem verses"][0:10]
```

--------------------------------

### Analyze Bait Input

Source: https://github.com/arbml/qawafi/blob/main/README.md

Analyze a diacritized input file using the BaitAnalysis module.

```python
from qawafi_server.bait_analysis import BaitAnalysis

analysis = BaitAnalysis()
output = analysis.analyze(read_from_path='baits_input.txt', override_tashkeel=True)
```

--------------------------------

### Display Theme Labels

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/theme.ipynb

Displays the mapping of theme labels to their respective integer IDs.

```python
theme2labels
```

--------------------------------

### Display Model Summary

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Prints a summary of the Keras model, detailing its layers, output shapes, and parameter counts.

```python
model.summary()
```

--------------------------------

### Split Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Generating train, test, and validation splits.

```python
split_filtered = generate_splits(filtered)
```

--------------------------------

### Load MetRecV2 Dataset

Source: https://context7.com/arbml/qawafi/llms.txt

Load the MetRecV2 dataset, which is specifically curated for training Arabic poetry meter classification models. Use the datasets.load_dataset function.

```python
from datasets import load_dataset

# Load full training dataset
dataset = load_dataset("Zaid/metrecv2", "train_all")
```

--------------------------------

### Define Model Architecture and Callbacks

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Constructs a bidirectional GRU model and configures model checkpointing for training.

```python
import tensorflow as tf
import numpy as np
import os
import time
import glob
from tensorflow.keras.layers import GRU, Embedding, Dense, Input, Dropout, Bidirectional, BatchNormalization, Flatten, Reshape
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import Tokenizer, text_to_word_sequence
from tensorflow.keras.preprocessing.sequence import pad_sequences
from sklearn.utils import shuffle
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt



def create_model():
  model = Sequential()
  model.add(Input((max_tokens,)))
  model.add(Embedding(10_000, 128))
  model.add(Bidirectional(GRU(units = 64, return_sequences=True, dropout=0.3)))
  model.add(Bidirectional(GRU(units = 64, return_sequences=True, dropout=0.3)))
  model.add(Bidirectional(GRU(units = 64, dropout=0.3)))
  model.add(Dropout(0.3))
  model.add(Dense(64, activation = 'relu'))
  model.add(Dropout(0.3))
  model.add(Dense(4, activation = 'softmax'))
  model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy', metrics = ['accuracy'])
  return model


checkpoint_path = "/content/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

callbacks = [tf.keras.callbacks.ModelCheckpoint(checkpoint_path,
                                                 monitor='val_accuracy',
                                                 verbose=1,
                                                 save_weights_only=True,
                                                 save_best_only= True,
                                                 mode='max')]
```

--------------------------------

### Download and Extract Model

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Downloads a model archive (model.zip) from Google Drive using gdown and extracts its contents.

```python
import gdown
url = 'https://drive.google.com/uc?id=1TKEZqVapq80O_E-qB2zqxjBN1ox_Vwu1' 
gdown.cached_download(url,'model.zip', quiet=False, postprocess=gdown.extractall)
```

--------------------------------

### Define Model Architecture and Checkpoints

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/theme.ipynb

Configures a bidirectional GRU model and sets up Keras ModelCheckpoint callbacks for training.

```python
import tensorflow as tf
import numpy as np
import os
import time
import glob
from tensorflow.keras.layers import GRU, Embedding, Dense, Input, Dropout, Bidirectional, BatchNormalization, Flatten, Reshape
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import Tokenizer, text_to_word_sequence
from tensorflow.keras.preprocessing.sequence import pad_sequences
from sklearn.utils import shuffle
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt



def create_model():
  model = Sequential()
  model.add(Input((max_tokens,)))
  model.add(Embedding(10_000, 128))
  model.add(Bidirectional(GRU(units = 64, return_sequences=True, dropout=0.3)))
  model.add(Bidirectional(GRU(units = 64, return_sequences=True, dropout=0.3)))
  model.add(Bidirectional(GRU(units = 64, dropout=0.3)))
  model.add(Dropout(0.3))
  model.add(Dense(64, activation = 'relu'))
  model.add(Dropout(0.3))
  model.add(Dense(len(labels2theme), activation = 'softmax'))
  model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy', metrics = ['accuracy'])
  return model


checkpoint_path = "/content/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

callbacks = [tf.keras.callbacks.ModelCheckpoint(checkpoint_path,
                                                 monitor='val_accuracy',
                                                 verbose=1,
                                                 save_weights_only=True,
                                                 save_best_only= True,
                                                 mode='max')]
```

--------------------------------

### Train Model with Callbacks

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/theme.ipynb

Fits the model to the training data, using validation data and callbacks for monitoring and saving. Specify epochs and batch size.

```python
model.fit(x_train, y_train, validation_data = (x_valid, y_valid) , callbacks= callbacks, epochs = 5, shuffle = True, batch_size=128)
```

--------------------------------

### Inspect Augmented Data Sample

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Retrieves the last sample from the augmented dataset.

```python
X_aug[-1], y_aug[-1]
```

--------------------------------

### Iterate and Find Closest Baits for Test Data

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Iterates through the test dataset, prints the label and sentence for a specific class, and then finds and prints the closest baits (similar sentences) for that sentence.

```python
cnt = 0 
for idx in range(len(dataset['test']['text'])):
  sample_sentence = dataset['test']['text'][idx]
  if dataset['test']['label'][idx] == cnt:
    print(labels[dataset['test']['label'][idx]], dataset['test']['label'][idx])
    print(sample_sentence)
    print(get_closest_baits(sample_sentence))
    cnt += 1
    break
```

--------------------------------

### Initialize Diacritization Tester

Source: https://github.com/arbml/qawafi/blob/main/diac_inference.ipynb

Loads the model configuration and initializes the tester instance for inference.

```python
from predict import DiacritizationTester
tester = DiacritizationTester('config/test.yml', 'cbhg')
```

--------------------------------

### Analyze Arabic Poetry via REST API

Source: https://context7.com/arbml/qawafi/llms.txt

Send a POST request to the /api/analyze endpoint with Arabic poetry text to receive comprehensive analysis results, including diacritization, meter, and theme.

```bash
# POST request to analyze Arabic poetry
curl -X POST "http://localhost:8000/api/analyze" \
  -H "Content-Type: multipart/form-data" \
  -F "baits=القلب أعلم يا عذول بدائه
وأحق منك بجفنه وبمائه
لا تعذل المشتاق في أشواقه
حتى يكون حشاك في أحشائه"
```

--------------------------------

### Load Expanded Tashkeela Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Loads the 'expanded_tashkeela' dataset using the Hugging Face 'datasets' library. This dataset contains Arabic text with diacritization.

```python
from datasets import load_dataset 
dataset = load_dataset('expanded_tashkeela')
```

--------------------------------

### Clean Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Applies Tnkeeh cleaning to a Hugging Face dataset.

```python
def clean_dataset(dataset):
    
    cleaner = tn.Tnkeeh(**data_args)
    dataset = cleaner.clean_hf_dataset(dataset, 'text')
    return dataset
```

--------------------------------

### Find k Closest Baits

Source: https://context7.com/arbml/qawafi/llms.txt

Use the analysis module to find the k most similar baits (verses) from a corpus to a given input. Returns a list of (bait_text, similarity_score) tuples.

```python
closest = analysis.get_closest_baits(baits, k=3)
```

--------------------------------

### Display Dataset Information

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Displays the structure and number of rows in the loaded dataset. This is useful for verifying that the dataset has been loaded correctly.

```python
dataset
```

--------------------------------

### Split Data for Training

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Splits the augmented dataset into training and validation sets.

```python
X_train, X_valid , y_train, y_valid = train_test_split(X_aug, y_aug, test_size = 0.15, random_state = 41)
```

--------------------------------

### Create Hugging Face Dataset and Split

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Converts a Pandas DataFrame into a Hugging Face Dataset object and splits it into training and testing sets. Removes unnecessary index columns.

```python
from datasets import Dataset
dataset = Dataset.from_pandas(df[['text', 'label']]).remove_columns(['__index_level_0__', '__index_level_1__']).train_test_split(test_size=0.1)
```

--------------------------------

### Analyze Poetry Samples via API

Source: https://github.com/arbml/qawafi/blob/main/qawafi_server/tests.ipynb

Iterates through poetry samples, formats them into a payload, and sends them to a remote API for analysis. Handles API responses and errors.

```python
for i, qaseeda in enumerate(samples):
    pprint(qaseeda)
    payload = ""
    for bait in qaseeda:
        payload += "\n".join(bait.split("#"))
        payload += "\n"
    print("*" * 80)
    print(payload)
    print("*" * 80)
    response = requests.post(
        "http://34.123.169.118:8000/api/analyze",
        data={\"baits\": payload},
    )
    time.sleep(5)
    if response.ok:
        pprint(response.json())
    else:
        print(i)
        raise('error')
```

--------------------------------

### Balance Dataset Classes

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Calculates sample limits per class and performs downsampling to balance the dataset.

```python
sample_amounts = {idx:50_000 if cnt > 50_000 else cnt for idx, cnt in has_accepted_era_five_baits_df['label'].value_counts().iteritems()}
print(sample_amounts)
has_accepted_era_five_baits_df = (
    has_accepted_era_five_baits_df.groupby('label').apply(lambda g: g.sample(
        # lookup number of samples to take
        n=sample_amounts[g.name],
        # enable replacement if len is less than number of samples expected
        replace=False  
    ))
)
```

--------------------------------

### Load JSON Dataset

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Loads a dataset from a JSON file using the Hugging Face datasets library.

```python
from datasets import load_dataset
dt = load_dataset('json',data_files=['/content/era_dataset.json'],field='data')
dt
```

--------------------------------

### Check Tashkeel

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Validating tashkeel percentage for a given string.

```python
check_percentage_tashkeel('أَصبَحَ المُلك لِلَّذي فَطر الخَل')
```

--------------------------------

### Check Diacritization Percentage for a Sample

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Applies the `check_percentage_tashkeel` function to a specific sample of Arabic text to determine if it meets the diacritization threshold. This is a manual check for a single text string.

```python
check_percentage_tashkeel('ْقَاضِي أَبُو إِسْحَاقَ ، فَاسْتَخْرَجَ دُرَرَهَا ، وَاسْتَحْلَبَ دِرَرَهَا ، وَإِنْ كَانَ قَدْ غَيَرَ أَسَانِيدَهَا لَقَدْ رَبَطَ مَعَاقِدَهَا ، وَلَمْ يَأْتِ بَعْدَهُمَا مَنْ يَلْحَقُ بِهِمَا .وَلَمَا مَنَ اللَهُ سُبْحَانَهُ بِالِاسْتِبْصَارِ فِي اسْتِثَارَةِ الْعُلُومِ مِنْ الْكِتَابِ الْعَزِيزِ حَسْبَ مَا مَهَدَتْهُ لَنَا الْمَشْيَخَةُ الَذِينَ لَقِينَا ، نَظَرْنَاهَا مِنْ ذَلِكَ الْمَطْرَحِ ، ثُمَ عَرَضْنَاهَا عَلَى مَا جَلَبَهُ الْعُلَمَاءُ ، وَسَبَرْنَاهَا بِعِيَارِ الْأَشْيَاخِ .فَمَا ات')
```

--------------------------------

### Load Ashaar Dataset

Source: https://context7.com/arbml/qawafi/llms.txt

Load the comprehensive Ashaar dataset, containing over 250,000 Arabic poems, using the HuggingFace Datasets library. Access poem text, poet, meter, and theme information.

```python
import datasets

# Load the full Ashaar dataset
ashaar = datasets.load_dataset('arbml/ashaar')

# Dataset structure:
# DatasetDict({
#     train: Dataset({
#         features: ['poem_text', 'poet', 'meter', 'theme', ...],
#         num_rows: 254630
#     })
# })

# Access specific poems
poem = ashaar['train'][0]
print(poem['poem_text'])

# Dataset statistics:
# - 254,630 poems
# - 3,857,429 baits (verses)
# - 7,167 poets
```

--------------------------------

### Load Labels

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Reads labels from a text file and removes newline characters.

```python
with open('labels.txt', 'r') as f:
  label2name = f.readlines()
  label2name = [name.replace('\n', '') for name in label2name]
```

--------------------------------

### Process Dataframe

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/diacratization.ipynb

Manipulating pandas dataframes for dataset preparation.

```python
df = ashaar['train'].to_pandas()
```

```python
df = df.explode('poem verses')
```

```python
df = df[['poem verses']]
df = df.dropna()
```

```python
df
```

--------------------------------

### Visualize Test Data Distribution

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Generate a count plot for the test labels.

```python
sns.countplot(y_test)
```

--------------------------------

### Define Arabic Meter Labels

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

A list defining the names of different Arabic poetic meters.

```python
labels = ['saree','kamel','mutakareb','mutadarak','munsareh','madeed','mujtath','ramal','baseet','khafeef','taweel','wafer','hazaj','rajaz','mudhare','muqtadheb','prose']
```

--------------------------------

### BaitAnalysis.predict_theme

Source: https://context7.com/arbml/qawafi/llms.txt

Categorizes the emotional theme of an Arabic poem.

```APIDOC
## POST /bait_analysis/predict_theme

### Description
Categorizes the poem by emotional theme using deep learning classification.

### Parameters
#### Request Body
- **poem** (string) - Required - The poem text.
- **max_tokens** (integer) - Optional - Maximum token limit for the model.

### Response
#### Success Response (200)
- **theme_labels** (list of strings) - Predicted theme labels.
```

--------------------------------

### Define Character Vocabulary and Mapping

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/embedding.ipynb

Creates a character-to-index mapping for Arabic text, including common characters and diacritics.

```python
vocab = list('إةابتثجحخدذرزسشصضطظعغفقكلمنهويىأءئؤ#آ ')
vocab += list('ًٌٍَُِّ') +['ْ']+['ٓ']
char2idx = {u:i+1 for i, u in enumerate(vocab)}
```

--------------------------------

### Inspect Era Labels

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/era.ipynb

Displays the mapping of historical eras to integer labels.

```python
era2labels
```

--------------------------------

### Set Random Seed

Source: https://github.com/arbml/qawafi/blob/main/Notebooks/meter.ipynb

Initializes the random number generator for reproducibility.

```python
import random
random.seed(400)
```