### Install tnkeeh Source: https://github.com/arbml/tnkeeh/blob/master/README.md Use pip to install the library. ```bash pip install tnkeeh ``` -------------------------------- ### Remove Repeated Characters Source: https://context7.com/arbml/tnkeeh/llms.txt Reduces sequences of repeated characters to a single instance. For example, 'هههههههه' becomes 'هه'. ```python import tnkeeh as tn # Remove repeated characters text = "هههههههه" clean = tn._remove_repeated_chars(text) ``` -------------------------------- ### Initialize Tnkeeh and Clean Dataset Source: https://github.com/arbml/tnkeeh/blob/master/Demo.ipynb Import the library and load a dataset, then initialize the cleaner and process the text column. ```python import tnkeeh as tn from datasets import load_dataset ``` ```python dataset = load_dataset('metrec') ``` ```python dataset ``` ```python cleander = tn.Tnkeeh(remove_diacritics = True) cleaned_dataset = cleander.clean_hf_dataset(dataset, 'text') ``` ```python print(cleaned_dataset['train'][0]['text']) print(dataset['train'][0]['text']) ``` -------------------------------- ### Initialize and use the Tnkeeh class for reusable pipelines Source: https://context7.com/arbml/tnkeeh/llms.txt The Tnkeeh class allows for configuring a persistent cleaning pipeline that can be applied to various data formats including HuggingFace datasets, raw strings, and pandas DataFrames. ```python import tnkeeh as tn from datasets import load_dataset # Initialize cleaner with desired preprocessing options cleaner = tn.Tnkeeh( remove_diacritics=True, # Remove Arabic diacritical marks remove_special_chars=True, # Remove special characters remove_tatweel=True, # Remove tatweel character (ـ) normalize=True, # Normalize digit encodings remove_english=False, # Keep English characters remove_repeated_chars=True, # Remove repeated characters (ههههه -> هه) remove_html_elements=True, # Strip HTML tags remove_links=True, # Remove URLs remove_twitter_meta=True, # Remove @mentions and #hashtags remove_long_words=False, # Keep words longer than 15 chars excluded_chars=['+', '-'] # Characters to preserve during special char removal ) # Clean a HuggingFace dataset dataset = load_dataset('metrec') cleaned_dataset = cleaner.clean_hf_dataset(dataset, 'text') # Output: Dataset with 'text' field cleaned according to configured options # Clean raw text directly raw_text = "وَأفْجَـعُ مَن فَقَدْنَا" cleaned_text = cleaner.clean_raw_text(raw_text) # Output: "وأفجـع من فقدنا" # Clean a pandas DataFrame import pandas as pd df = pd.DataFrame({'content': ['السلام عليكم', 'كيًف حالكم']}) cleaned_df = cleaner.clean_data_frame(df, 'content') # Output: DataFrame with cleaned 'content' column ``` -------------------------------- ### Clean HuggingFace Datasets Source: https://github.com/arbml/tnkeeh/blob/master/README.md Clean a HuggingFace dataset using the Tnkeeh class. ```python import tnkeeh as tn from datasets import load_dataset dataset = load_dataset('metrec') cleaner = tn.Tnkeeh(remove_diacritics = True) cleaned_dataset = cleaner.clean_hf_dataset(dataset, 'text') ``` -------------------------------- ### Tnkeeh Class - Reusable Text Cleaner Source: https://context7.com/arbml/tnkeeh/llms.txt The Tnkeeh class allows for the creation of reusable preprocessing pipelines. It can be initialized with various options to clean HuggingFace datasets, text files, raw text strings, and pandas DataFrames. ```APIDOC ## Tnkeeh Class - Reusable Text Cleaner ### Description The `Tnkeeh` class provides a reusable preprocessing pipeline that can be configured once and applied to multiple datasets. It supports cleaning HuggingFace datasets, text files, raw text strings, and pandas DataFrames with consistent preprocessing options. ### Initialization ```python import tnkeeh as tn cleaner = tn.Tnkeeh( remove_diacritics=True, # Remove Arabic diacritical marks remove_special_chars=True, # Remove special characters remove_tatweel=True, # Remove tatweel character (ـ) normalize=True, # Normalize digit encodings remove_english=False, # Keep English characters remove_repeated_chars=True, # Remove repeated characters (ههههه -> هه) remove_html_elements=True, # Strip HTML tags remove_links=True, # Remove URLs remove_twitter_meta=True, # Remove @mentions and #hashtags remove_long_words=False, # Keep words longer than 15 chars excluded_chars=['+', '-'] # Characters to preserve during special char removal ) ``` ### Methods #### `clean_hf_dataset(dataset, text_column, **kwargs)` Cleans a HuggingFace dataset. - **dataset** (DatasetDict) - The HuggingFace dataset to clean. - **text_column** (str) - The name of the column containing text to clean. - **kwargs** - Additional arguments passed to the `Tnkeeh` constructor. ```python from datasets import load_dataset dataset = load_dataset('metrec') cleaned_dataset = cleaner.clean_hf_dataset(dataset, 'text') ``` #### `clean_raw_text(text)` Cleans a raw text string. - **text** (str) - The raw text string to clean. ```python raw_text = "وَأفْجَـعُ مَن فَقَدْنَا" cleaned_text = cleaner.clean_raw_text(raw_text) # Output: "وأفجـع من فقدنا" ``` #### `clean_data_frame(df, text_column, **kwargs)` Cleans a pandas DataFrame. - **df** (pd.DataFrame) - The DataFrame to clean. - **text_column** (str) - The name of the column containing text to clean. - **kwargs** - Additional arguments passed to the `Tnkeeh` constructor. ```python import pandas as pd df = pd.DataFrame({'content': ['السلام عليكم', 'كيًف حالكم']}) cleaned_df = cleaner.clean_data_frame(df, 'content') ``` ``` -------------------------------- ### Split Raw Text Data Source: https://context7.com/arbml/tnkeeh/llms.txt Splits a raw text file into training and testing sets based on a specified ratio. It creates 'train.txt' and 'test.txt' files in a 'data' directory. ```python import tnkeeh as tn # Split raw text data (80% train, 20% test) tn.split_raw_data('corpus.txt', split_ratio=0.8) ``` -------------------------------- ### Data Splitting API Source: https://github.com/arbml/tnkeeh/blob/master/README.md Utilities for splitting raw, classification, or parallel data into training and testing sets. ```APIDOC ## POST /split_raw_data ### Description Splits raw data into training and testing sets. ### Parameters #### Request Body - **data_path** (string) - Required - Path to the raw data file. - **split_ratio** (float) - Required - Ratio for splitting (e.g., 0.8). ## POST /split_classification_data ### Description Splits data and labels into training and testing sets. ### Parameters #### Request Body - **data_path** (string) - Required - Path to the data file. - **lbls_path** (string) - Required - Path to the labels file. - **split_ratio** (float) - Required - Ratio for splitting. ## POST /split_parallel_data ### Description Splits input and target data, commonly used for translation tasks. ### Parameters #### Request Body - **ar_data_path** (string) - Required - Path to Arabic data. - **en_data_path** (string) - Required - Path to English data. ``` -------------------------------- ### Clean Data Source: https://github.com/arbml/tnkeeh/blob/master/README.md Clean a text file by specifying input and output paths. ```python import tnkeeh as tn tn.clean_data(file_path = 'data.txt', save_path = 'cleaned_data.txt',) ``` -------------------------------- ### Citation Source: https://github.com/arbml/tnkeeh/blob/master/README.md BibTeX citation for the library. ```bibtex @misc{tnkeeh2020, author = {Zaid Alyafeai and Maged Saeed}, title = {tkseem: A Preprocessing Library for Arabic.}, year = {2020}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/ARBML/tnkeeh}} } ``` -------------------------------- ### Data Splitting Functions Source: https://context7.com/arbml/tnkeeh/llms.txt Functions for splitting raw, classification, or parallel datasets into training and testing sets. ```APIDOC ## Data Splitting Functions ### Description Provides utilities to split datasets into training and testing sets based on specific data types (raw, classification, or parallel). ### Methods - **split_raw_data(file_path, split_ratio)**: Splits plain text files. - **split_classification_data(text_path, label_path, split_ratio)**: Splits labeled data. - **split_parallel_data(source_path, target_path, split_ratio)**: Splits translation pairs. ### Parameters - **split_ratio** (float) - Required - The ratio for the training set (e.g., 0.8 for 80%). ``` -------------------------------- ### Low-Level Text Processing Source: https://context7.com/arbml/tnkeeh/llms.txt Individual functions for fine-grained control over Arabic text preprocessing. ```APIDOC ## Low-Level Text Processing ### Description Exposes individual functions for specific text cleaning tasks such as removing diacritics, HTML, Twitter metadata, and extra spaces. ### Available Functions - **_remove_diacritics(text)**: Removes Arabic tashkeel. - **_remove_special_chars(text, excluded_chars)**: Removes special characters while optionally preserving specific ones. - **_remove_html_elements(text)**: Strips HTML tags. - **_remove_twitter_meta(text)**: Removes mentions, hashtags, and links. - **_remove_repeated_chars(text)**: Reduces repeated characters. - **_remove_extra_spaces(text)**: Normalizes whitespace. - **_remove_links(text)**: Removes URLs. ``` -------------------------------- ### Split Data Source: https://github.com/arbml/tnkeeh/blob/master/README.md Split raw, classification, or parallel data into training and testing sets. ```python import tnkeeh as tn tn.split_raw_data(data_path, split_ratio = 0.8) ``` ```python import tnkeeh as tn tn.split_classification_data(data_path, lbls_path, split_ratio = 0.8) ``` ```python tn.split_parallel_data('ar_data.txt','en_data.txt') ``` -------------------------------- ### clean_hf_dataset - HuggingFace Dataset Cleaning Source: https://context7.com/arbml/tnkeeh/llms.txt The `clean_hf_dataset` function applies preprocessing to HuggingFace datasets, cleaning a specified text field across all splits using the dataset's map function. ```APIDOC ## clean_hf_dataset - HuggingFace Dataset Cleaning ### Description The `clean_hf_dataset` function applies preprocessing to HuggingFace datasets, cleaning a specified text field across all splits (train, test, validation) using the dataset's map function for efficient processing. ### Parameters #### Path Parameters - **dataset** (DatasetDict) - Required - The HuggingFace dataset to clean. - **text_column** (str) - Required - The name of the column containing the text to be cleaned. #### Query Parameters - **remove_diacritics** (bool) - Optional - Remove Arabic diacritical marks. - **remove_special_chars** (bool) - Optional - Remove special characters. - **normalize** (bool) - Optional - Normalize digit encodings. - **remove_tatweel** (bool) - Optional - Remove tatweel character (ـ). - **remove_english** (bool) - Optional - Keep or remove English characters. - **remove_repeated_chars** (bool) - Optional - Remove repeated characters. - **remove_html_elements** (bool) - Optional - Strip HTML tags. - **remove_links** (bool) - Optional - Remove URLs. - **remove_twitter_meta** (bool) - Optional - Remove @mentions and #hashtags. - **remove_long_words** (bool) - Optional - Remove words longer than 15 characters. - **excluded_chars** (list[str]) - Optional - Characters to preserve during special char removal. ### Request Example ```python import tnkeeh as tn from datasets import load_dataset # Load an Arabic dataset dataset = load_dataset('metrec') # Clean the 'text' field with specified options cleaned_dataset = tn.clean_hf_dataset( dataset, 'text', remove_diacritics=True, remove_special_chars=True, normalize=True ) # Compare original vs cleaned print(dataset['train'][0]['text']) # Output: "يا ضيفَ طيفٍ ما هَداهُ لمَضجَعي # إلا لهيبٌ في الحشى يتوقّدُ" print(cleaned_dataset['train'][0]['text']) # Output: "يا ضيف طيف ما هداه لمضجعي # إلا لهيب في الحشى يتوقد" ``` ### Response Returns a `DatasetDict` with the specified text column cleaned across all splits. ``` -------------------------------- ### Clean HuggingFace Dataset with Translation Source: https://context7.com/arbml/tnkeeh/llms.txt Cleans a HuggingFace dataset by processing a specified text column. It can remove diacritics and translate the text to English using Helsinki-NLP models. ```python translated_dataset = tn.clean_hf_dataset( dataset, 'text', remove_diacritics=True, translate='en' # Translate Arabic to English ) ``` -------------------------------- ### Split Parallel Data for Translation Source: https://context7.com/arbml/tnkeeh/llms.txt Splits parallel text data (e.g., source and target language sentences) into training and testing sets. This is useful for machine translation tasks. ```python import tnkeeh as tn # Split parallel data for translation tasks tn.split_parallel_data( 'ar_data.txt', # Arabic source sentences 'en_data.txt', # English target sentences split_ratio=0.8 ) ``` -------------------------------- ### Read Data Source: https://github.com/arbml/tnkeeh/blob/master/README.md Read split data based on the specified mode. ```python import tnkeeh as tn train_data, test_data = tn.read_data(mode = 0) ``` -------------------------------- ### Split Classification Data Source: https://context7.com/arbml/tnkeeh/llms.txt Splits text data and their corresponding labels into training and testing sets. It generates separate files for training/testing data and labels. ```python import tnkeeh as tn # Split classification data with labels tn.split_classification_data( 'texts.txt', # One text per line 'labels.txt', # Corresponding labels split_ratio=0.8 ) ``` -------------------------------- ### Read Split Data Source: https://context7.com/arbml/tnkeeh/llms.txt Reads data that has been previously split using the tnkeeh splitting functions. Supports different modes for raw, classification, and parallel data. ```python import tnkeeh as tn # Read split data back train_data, test_data = tn.read_data(mode=0) # Raw data train_data, test_data, train_lbls, test_lbls = tn.read_data(mode=1) # Classification train_inp, train_tar, test_inp, test_tar = tn.read_data(mode=2) # Parallel ``` -------------------------------- ### Clean HuggingFace datasets using clean_hf_dataset Source: https://context7.com/arbml/tnkeeh/llms.txt The clean_hf_dataset function applies cleaning operations to a specific text field across all splits of a HuggingFace DatasetDict. ```python import tnkeeh as tn from datasets import load_dataset # Load an Arabic dataset dataset = load_dataset('metrec') # DatasetDict with train (47124 rows) and test (8316 rows) splits # Clean the 'text' field with diacritics removal cleaned_dataset = tn.clean_hf_dataset( dataset, 'text', remove_diacritics=True, remove_special_chars=True, normalize=True ) # Compare original vs cleaned print(dataset['train'][0]['text']) # Output: "يا ضيفَ طيفٍ ما هَداهُ لمَضجَعي # إلا لهيبٌ في الحشى يتوقّدُ" print(cleaned_dataset['train'][0]['text']) # Output: "يا ضيف طيف ما هداه لمضجعي # إلا لهيب في الحشى يتوقد" ``` -------------------------------- ### Clean text files using clean_data Source: https://context7.com/arbml/tnkeeh/llms.txt The clean_data function processes text files directly, supporting chunked processing for large files to optimize memory usage. ```python import tnkeeh as tn # Basic file cleaning with diacritics removal tn.clean_data( file_path='arabic_corpus.txt', save_path='cleaned_corpus.txt', remove_diacritics=True, remove_special_chars=True, normalize=True ) # Output: Cleaned text saved to 'cleaned_corpus.txt' # Process large files in chunks tn.clean_data( file_path='large_corpus.txt', save_path='cleaned_large.txt', remove_diacritics=True, remove_html_elements=True, remove_links=True, by_chunk=True, chunk_size=100000 # Process 100k lines at a time ) # Output: Multiple files: cleaned_large_0.txt, cleaned_large_1.txt, etc. # Full preprocessing pipeline for social media data tn.clean_data( file_path='tweets.txt', save_path='cleaned_tweets.txt', remove_diacritics=True, remove_special_chars=True, remove_twitter_meta=True, # Remove @mentions and #hashtags remove_links=True, remove_repeated_chars=True, # Handle expressions like "هههههه" remove_long_words=True # Remove words > 15 characters ) ``` -------------------------------- ### clean_data_frame Source: https://context7.com/arbml/tnkeeh/llms.txt Applies Arabic text preprocessing to a specified column in a pandas DataFrame. ```APIDOC ## clean_data_frame ### Description Applies Arabic text preprocessing to a specified column in a pandas DataFrame, returning the modified DataFrame with cleaned text. ### Parameters #### Request Body - **df** (DataFrame) - Required - The pandas DataFrame containing the text column. - **column_name** (str) - Required - The name of the column to clean. - **remove_diacritics** (bool) - Optional - Whether to remove Arabic diacritics. - **remove_special_chars** (bool) - Optional - Whether to remove special characters. ### Request Example { "df": "pandas_dataframe_object", "column_name": "text", "remove_diacritics": true, "remove_special_chars": true } ``` -------------------------------- ### clean_data - File-Based Text Cleaning Source: https://context7.com/arbml/tnkeeh/llms.txt The `clean_data` function processes text files directly, applying various cleaning operations and saving results to a new file. It supports chunked processing for large files. ```APIDOC ## clean_data - File-Based Text Cleaning ### Description The `clean_data` function processes text files directly, applying various cleaning operations and saving results to a new file. It supports chunked processing for large files to manage memory efficiently. ### Parameters #### Path Parameters - **file_path** (str) - Required - Path to the input text file. - **save_path** (str) - Required - Path to save the cleaned text file. #### Query Parameters - **remove_diacritics** (bool) - Optional - Remove Arabic diacritical marks. - **remove_special_chars** (bool) - Optional - Remove special characters. - **normalize** (bool) - Optional - Normalize digit encodings. - **remove_html_elements** (bool) - Optional - Strip HTML tags. - **remove_links** (bool) - Optional - Remove URLs. - **remove_twitter_meta** (bool) - Optional - Remove @mentions and #hashtags. - **remove_repeated_chars** (bool) - Optional - Remove repeated characters. - **remove_long_words** (bool) - Optional - Remove words longer than 15 characters. - **by_chunk** (bool) - Optional - Process the file in chunks (for large files). - **chunk_size** (int) - Optional - Number of lines to process per chunk when `by_chunk` is True. ### Request Example ```python import tnkeeh as tn # Basic file cleaning tn.clean_data( file_path='arabic_corpus.txt', save_path='cleaned_corpus.txt', remove_diacritics=True, remove_special_chars=True, normalize=True ) # Process large files in chunks tn.clean_data( file_path='large_corpus.txt', save_path='cleaned_large.txt', remove_diacritics=True, remove_html_elements=True, remove_links=True, by_chunk=True, chunk_size=100000 ) # Full preprocessing pipeline for social media data tn.clean_data( file_path='tweets.txt', save_path='cleaned_tweets.txt', remove_diacritics=True, remove_special_chars=True, remove_twitter_meta=True, remove_links=True, remove_repeated_chars=True, remove_long_words=True ) ``` ### Response Cleaned text is saved to the specified `save_path`. If `by_chunk` is True, multiple files named like `cleaned_large_0.txt`, `cleaned_large_1.txt`, etc., will be created. ``` -------------------------------- ### Remove URLs Source: https://context7.com/arbml/tnkeeh/llms.txt Removes all URLs (http, https) from a given text string. ```python import tnkeeh as tn # Remove URLs/links text = "زوروا موقعنا http://example.com للمزيد" clean = tn._remove_links(text) ``` -------------------------------- ### Data Cleaning API Source: https://github.com/arbml/tnkeeh/blob/master/README.md Cleans Arabic text files by applying various filters such as diacritic removal, special character removal, and normalization. ```APIDOC ## POST /clean_data ### Description Cleans a text file based on specified cleaning parameters and saves the output to a new file. ### Method POST ### Parameters #### Request Body - **file_path** (string) - Required - Path to the input text file. - **save_path** (string) - Required - Path to save the cleaned text file. - **segment** (boolean) - Optional - Uses farasa for segmentation. - **remove_diacritics** (boolean) - Optional - Removes all diacritics. - **remove_special_chars** (boolean) - Optional - Removes all special characters. - **remove_english** (boolean) - Optional - Removes English alphabets and digits. - **normalize** (boolean) - Optional - Matches digits with different encodings. - **remove_tatweel** (boolean) - Optional - Removes the tatweel character. - **remove_repeated_chars** (boolean) - Optional - Removes characters appearing three times in sequence. - **remove_html_elements** (boolean) - Optional - Removes HTML elements. - **remove_links** (boolean) - Optional - Removes URLs. - **remove_twitter_meta** (boolean) - Optional - Removes Twitter mentions, links, and hashtags. - **remove_long_words** (boolean) - Optional - Removes words longer than 15 characters. - **by_chunk** (boolean) - Optional - Reads files by chunks. - **chunk_size** (integer) - Optional - Size of chunks if by_chunk is enabled. ``` -------------------------------- ### Remove HTML Elements Source: https://context7.com/arbml/tnkeeh/llms.txt Strips all HTML tags and their content from a given text string. ```python import tnkeeh as tn # Remove HTML elements text = '' clean = tn._remove_html_elements(text) ``` -------------------------------- ### Remove Arabic Diacritics Source: https://context7.com/arbml/tnkeeh/llms.txt Removes diacritical marks (tashkeel) from Arabic text. ```python import tnkeeh as tn # Remove Arabic diacritics (tashkeel) text = "وَأفْجَـعُ مَن فَقَدْنَا مَن وَّجَدْنَا" clean = tn._remove_diacritics(text) ``` -------------------------------- ### Remove Extra Spaces Source: https://context7.com/arbml/tnkeeh/llms.txt Replaces multiple consecutive whitespace characters with a single space. ```python import tnkeeh as tn # Remove extra spaces text = "اهلا كيف حالك" clean = tn._remove_extra_spaces(text) ``` -------------------------------- ### Remove Special Characters with Exclusions Source: https://context7.com/arbml/tnkeeh/llms.txt Removes special characters from text while allowing specific characters to be preserved. Useful for keeping mathematical operators or date separators. ```python import tnkeeh as tn # Remove special chars with exclusions text = "3 + 3 = 6 and 9/8/1770" clean = tn._remove_special_chars(text, excluded_chars=['+', '=', '/']) ``` -------------------------------- ### Remove Special Characters Source: https://context7.com/arbml/tnkeeh/llms.txt Removes special characters from text, preserving Arabic letters, English letters, and digits. By default, it removes most punctuation and symbols. ```python import tnkeeh as tn # Remove special characters (preserving Arabic, English, digits) text = "كيف حالكم ، يا أشقاء!" clean = tn._remove_special_chars(text) ``` -------------------------------- ### Clean Pandas DataFrame with Arabic Text Source: https://context7.com/arbml/tnkeeh/llms.txt Applies Arabic text preprocessing to a specified column in a pandas DataFrame. This function can remove diacritics and special characters. ```python import tnkeeh as tn import pandas as pd # Create DataFrame with Arabic text df = pd.DataFrame({ 'id': [1, 2, 3], 'text': [ 'السلام عليكم', 'كيًف حالكم يا أصدقاء؟', 'هذا نصٌّ عربيٌّ مُشَكَّل' ] }) # Clean the text column cleaned_df = tn.clean_data_frame( df, 'text', remove_diacritics=True, remove_special_chars=True ) print(cleaned_df) ``` -------------------------------- ### Remove Twitter Metadata Source: https://context7.com/arbml/tnkeeh/llms.txt Removes common Twitter metadata elements such as mentions (@username), hashtags (#topic), and URLs from a text string. ```python import tnkeeh as tn # Remove Twitter metadata (mentions, hashtags, links) text = "@arthurlacoste check this link: https://t.co/abc ! so #nsfw" clean = tn._remove_twitter_meta(text) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.