### Python Full Normalization Example Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb Demonstrates full text normalization using a predefined list of converters including dimension preprocessing, unit normalization, number to word conversion, symbol conversion, ordinal normalization, and Turkish lowercasing. It iterates through example texts, applies the normalization, and prints the original and normalized versions. ```python print("\nFull normalization example:") full_converters = [ preprocess_dimensions, normalize_dimensions, normalize_units, convert_numbers_to_words_wrapper, convert_symbols, normalize_ordinals, turkish_lower ] for text in example_texts[3:]: print(f"\nOriginal: {text}") normalized = normalize(text, full_converters) print(f"Normalized: {normalized}") ``` -------------------------------- ### Get Available Turkish Normalizers Source: https://github.com/ysdede/trnorm/blob/master/README_SIMPLIFIED_NORMALIZER.md Shows how to retrieve a list of all available normalization functions within the 'trnorm' package. Each normalizer is presented with a brief explanation of its purpose. ```python from trnorm import get_available_normalizers normalizers = get_available_normalizers() for name, description in normalizers.items(): print(f"{name}: {description}") ``` -------------------------------- ### Selective Normalization Steps with `normalize` Function Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md Shows how to apply only specific normalization steps by passing boolean arguments to the `normalize` function. This example demonstrates converting numbers to text while preserving case and diacritics, and also applying legacy normalization. ```python from trnorm import normalize # Only convert numbers to text, keep case and diacritical marks text = "Âlim insanlar 15 kitap okumuş." normalized_text = normalize( text, apply_number_conversion=True, apply_ordinal_normalization=False, apply_symbol_conversion=False, apply_multiplication_symbol=False, apply_unit_normalization=False, apply_time_normalization=False, lowercase=False, remove_hats=False ) print(normalized_text) # Output: "Âlim insanlar on beş kitap okumuş." # Apply legacy normalization (more aggressive, removes punctuation) text = "Bugün 3x4 metre halı aldım." normalized_text = normalize(text, apply_legacy_normalization=True) print(normalized_text) # Output: "bugün üç çarpı dört metre halı aldım" ``` -------------------------------- ### Basic Text Similarity Metrics Example - Python Source: https://github.com/ysdede/trnorm/blob/master/docs/README_metrics.md Demonstrates the basic usage of WER, CER, and Levenshtein distance for evaluating similarity between two Turkish text strings. This example shows how to import and apply these metrics to assess differences in text. ```python from trnorm.metrics import wer, cer, levenshtein_distance reference = "Türkçe metin normalizasyonu önemlidir" hypothesis = "Türkçe metin normalizasyon önemli" # Calculate metrics wer_score = wer(reference, hypothesis) cer_score = cer(reference, hypothesis) lev_distance = levenshtein_distance(reference, hypothesis) print(f"WER: {wer_score:.4f}") print(f"CER: {cer_score:.4f}") print(f"Levenshtein Distance: {lev_distance}") ``` -------------------------------- ### Python Custom Normalization Combinations Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb Shows how to apply custom combinations of converters for specific normalization tasks. Examples include normalizing only dimensions, only numbers to words, only units, and a full normalization. ```python print("\nCustom converter combinations:") text = "Ürün boyutları 120x180cm ve fiyatı 1.250,75 TL'dir." print(f"\nOriginal: {text}") # Only dimension handling dimension_only = normalize(text, [preprocess_dimensions, normalize_dimensions]) print(f"Dimension only: {dimension_only}") # Only number conversion number_only = normalize(text, [convert_numbers_to_words_wrapper]) print(f"Number only: {number_only}") # Only unit normalization unit_only = normalize(text, [normalize_units]) print(f"Unit only: {unit_only}") # Full normalization full_norm = normalize(text, full_converters) print(f"Full normalization: {full_norm}") ``` -------------------------------- ### Demonstrate trnorm Normalization Functions in Python Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb This Python code showcases the usage of the `normalize` function from the trnorm library with different sets of converters. It demonstrates basic normalization including number-to-word conversion and lowercasing, as well as specialized normalization for dimensions and units. The code iterates through example texts, applies the specified converters, and prints both the original and normalized versions. ```python # Example texts with various normalization needs example_texts = [ "Bugün 15. kattaki 3 toplantıya katıldım.", "Saat 14:30'da %25 indirimli ürünler satışa çıkacak.", "II. Dünya Savaşı 1939-1945 yılları arasında gerçekleşti.", "Ürün fiyatı 1.250,75 TL'dir.", "1. sınıfta 23 öğrenci var.", "Dün 3x4 metre halı aldım.", "Âlim insanlar bilgilerini paylaşır.", ] # Examples for dimension and unit handling mixed_examples = [ "Odanın boyutları 2x3x4 metre.", "Halının boyutu 120x180cm.", "Ağırlığı 5 kg. ve uzunluğu 10 metre.", "Sıcaklık 25 °C ve nem %60.", ] # Basic normalization example print("Basic normalization example:") basic_converters = [ convert_numbers_to_words_wrapper, turkish_lower ] for text in example_texts[:3]: print(f"\nOriginal: {text}") normalized = normalize(text, basic_converters) print(f"Normalized: {normalized}") # Dimension and unit handling example print("\nDimension and unit handling example:") dimension_converters = [ preprocess_dimensions, normalize_dimensions, normalize_units, convert_numbers_to_words_wrapper, turkish_lower ] for text in mixed_examples[:2]: print(f"\nOriginal: {text}") normalized = normalize(text, dimension_converters) print(f"Normalized: {normalized}") ``` -------------------------------- ### Python Batch Normalization Example Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb Illustrates batch processing of a list of texts using basic converters. The `normalize` function is applied to multiple texts simultaneously, and the results are iterated and printed with numbering. ```python print("\nBatch processing example:") batch_normalized = normalize(example_texts, basic_converters) for i, normalized in enumerate(batch_normalized, 1): print(f"{i}. {normalized}") ``` -------------------------------- ### Get Available Transformers and Descriptions Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Retrieves all available transformers and their descriptions from the trnorm library. It iterates through the returned dictionary and prints each transformer's name and its corresponding description. ```python transformers = get_available_transformers() for name, description in transformers.items(): print(f"{name}: {description}") ``` -------------------------------- ### Python: Get Available Transformers in trnorm Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Demonstrates how to retrieve a list of all available transformers that can be used within the `trnorm` library's pipelines. This function is useful for inspecting the library's capabilities. ```python from trnorm import get_available_transformers ``` -------------------------------- ### Unit Abbreviation Conversion to Full Text Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md Provides examples of converting common unit abbreviations (e.g., 'cm', 'm') into their full Turkish text forms ('santimetre', 'metre'). It also covers scenarios with multiple units and disabling unit normalization. ```python from trnorm import normalize # Convert unit abbreviations to full text text = "Masanın yüksekliği 75 cm." normalized_text = normalize(text) print(normalized_text) # Output: "masanın yüksekliği yetmiş beş santimetre." # Multiple units in the same text text = "Odanın boyutları 5 m x 4 m, yüksekliği 3 m." normalized_text = normalize(text) print(normalized_text) # Output: "odanın boyutları beş metre çarpı dört metre, yüksekliği üç metre." # Disable unit normalization text = "Sıcaklık 25 °C." normalized_text = normalize(text, apply_unit_normalization=False) print(normalized_text) # Output: "sıcaklık yirmi beş °c." ``` -------------------------------- ### Apply Custom Normalization Pipeline in Python Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb This Python code defines a custom normalization pipeline using functions from the trnorm library. It then applies this pipeline to a list of example Turkish texts and prints the normalized output. The pipeline includes functions for time normalization, apostrophe removal, specific character handling (sapkasiz), dimension preprocessing, symbol conversion, and number-to-word conversion. ```python # print(remove_apostrophes("Turner'ın 'Köle Gemisi' isimli tablosuna bakıyoruz.")) # print(remove_apostrophes("Turner'ın Köle Gemisi isimli tablosuna bakıyoruz.")) # print(remove_apostrophes("""Turner"ın "Köle Gemisi"isimli tablosuna bakıyoruz."""")) # print(remove_apostrophes("""Turner"ın Köle Gemisi isimli tablosuna bakıyoruz."""")) # print(remove_apostrophes("Turner'ın (Köle Gemisi) isimli tablosuna bakıyoruz.")) # print(remove_apostrophes("Turner'ın Köle Gemisi isimli tablosuna bakıyoruz.")) # print(remove_apostrophes('"Turner"ın "Köle Gemisi"isimli tablosuna bakıyoruz."')) # print(remove_apostrophes('"'Turner"ın Köle Gemisi isimli tablosuna bakıyoruz.'"')) # print() # my_pipeline = [remove_apostrophes, normalize_ordinals, preprocess_dimensions, convert_numbers_to_words_wrapper] my_pipeline = [normalize_times, remove_apostrophes, sapkasiz, preprocess_dimensions, convert_symbols, convert_numbers_to_words_wrapper] # Changed order def apply_normalizers(text): for norm in my_pipeline: text = norm(text) return text example_texts = [ "Turner'ın 'Köle Gemisi' isimli tablosuna bakıyoruz.", "Odanın boyutları 2x3x4 metre.", "Halının boyutu 120x180cm.", "Ağırlığı 5 kg. ve uzunluğu 10 metre.", "Sıcaklık 25 °C ve nem %60.", "Saat 14:30'da %25 indirimli ürünler satışa çıkacak.", "Ürün fiyatı 1.250,75 TL'dir.", "1. sınıfta 23 öğrenci var.", "Dün 3x4 metre halı aldım.", "II. Wilhelm, Almanya imparatoruydu.", "V. Karl, Kutsal Roma İmparatoru'ydu.", "1'inci sınıf 1inci kat", "2. Mahmud ya da II. Mahmud. BUG", "1.000 ₺", "$500", "600 $", "kâğıt ve kalem", "Âdem ile Havva", "îmanın kimde", "Îman büyük harfle şapkalı", "hûr ne", "Yavru ile kâtip" ] for e in example_texts: print(apply_normalizers(e)) ``` -------------------------------- ### Integration of Metrics with Normalization - Python Source: https://github.com/ysdede/trnorm/blob/master/docs/README_metrics.md Illustrates how the metrics module can be integrated with other trnorm modules, specifically `convert_numbers_to_words_wrapper`. This example shows how text normalization (converting numbers to words) can potentially improve WER scores when comparing ASR output against a reference. ```python from trnorm.metrics import wer from trnorm.num_to_text import convert_numbers_to_words_wrapper # Original text with numbers reference = "Bu 42 sayısı önemlidir" # Text with normalized numbers normalized = convert_numbers_to_words_wrapper(reference) # ASR output hypothesis = "bu kırk iki sayısı önemli" # Calculate WER between original and ASR output original_wer = wer(reference, hypothesis) # Calculate WER between normalized and ASR output normalized_wer = wer(normalized, hypothesis) print(f"Original WER: {original_wer:.4f}") print(f"Normalized WER: {normalized_wer:.4f}") ``` -------------------------------- ### Process ASR Logs with WER and CER - Python Source: https://github.com/ysdede/trnorm/blob/master/docs/README_metrics.md An example script snippet for processing Automatic Speech Recognition (ASR) log files. It iterates through log entries, extracts reference and hypothesis texts, and calculates WER and CER scores for each entry to evaluate ASR performance. ```python from trnorm.metrics import wer, cer # For each line in an ASR log file for line in asr_log: reference = line["reference_text"] hypothesis = line["asr_output"] # Calculate metrics wer_score = wer(reference, hypothesis) cer_score = cer(reference, hypothesis) # Process or store results print(f"WER: {wer_score:.4f}, CER: {cer_score:.4f}") ``` -------------------------------- ### Python: Create Custom Conversion Function Source: https://github.com/ysdede/trnorm/blob/master/README_BASIC_NORMALIZER.md Illustrates how to define and integrate a custom conversion function into the normalization process. The example shows a function to replace Turkish month names with their numeric representations, which can then be added to the list of converters used by `normalize`. ```python from trnorm import normalize from trnorm.num_to_text import convert_numbers_to_words_wrapper from trnorm.legacy_normalizer import turkish_lower def replace_turkish_month_names(text): """Replace Turkish month names with their numeric representation.""" month_mapping = { "ocak": "1", "şubat": "2", # ... other months } result = text for month, number in month_mapping.items(): result = result.replace(month, number) result = result.replace(month.capitalize(), number) return result # Add your custom function to the converter list custom_converters = [ replace_turkish_month_names, convert_numbers_to_words_wrapper, turkish_lower ] normalized_text = normalize("5 Ocak 2023 tarihinde toplantı var.", custom_converters) # Result: "beş bir iki bin yirmi üç tarihinde toplantı var." ``` -------------------------------- ### Python Unit Test Execution for Turkish Text Utilities Source: https://github.com/ysdede/trnorm/blob/master/docs/README_text_utils.md Shows how to run the unit tests for the Turkish Text Utilities module using Python's unittest framework. This command executes tests defined in 'test_text_utils.py'. ```python python -m unittest test_text_utils.py ``` -------------------------------- ### TransformerPipeline Class Initialization and Apply Method Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Provides the API reference for the TransformerPipeline class, detailing its constructor which accepts an optional list of transformer names, and its 'apply' method for processing text. ```python from typing import Optional, List, Union class TransformerPipeline: def __init__(self, transformers: Optional[List[str]] = None): """Initialize with list of transformer names""" pass def apply(self, text: Union[str, List[str]]) -> Union[str, List[str]]: """Apply all transformers in sequence""" pass ``` -------------------------------- ### Import trnorm Utilities in Python Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb This snippet demonstrates importing various normalization functions from the trnorm package. It includes utilities for dimensions, units, numbers to words, ordinals, symbols, apostrophe handling, and text manipulation. ```python import sys import os # Add the parent directory to the path to import the trnorm package # sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) from trnorm import normalize from trnorm.dimension_utils import preprocess_dimensions, normalize_dimensions from trnorm.unit_utils import normalize_units from trnorm.num_to_text import convert_numbers_to_words_wrapper from trnorm.ordinals import normalize_ordinals from trnorm.symbols import convert_symbols from trnorm.apostrophe_handler import remove_apostrophes from trnorm.text_utils import ( turkish_lower, turkish_upper, turkish_capitalize, is_turkish_upper, sapkasiz ) from trnorm.time_utils import normalize_times from trnorm.symbols import SymbolConverter, convert_symbols, add_symbol_mapping from trnorm.symbol_mappings import get_all_mappings ``` -------------------------------- ### Initialize and Apply Transformer (Python) Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md This Python code defines a Transformer class. The `__init__` method initializes the transformer with a name, a transformation function, a description, and optional keyword arguments. The `__call__` method applies the defined transformation to an input string, returning the transformed string. ```python class Transformer: def __init__(self, name: str, func: TransformerFunc, description: str, **kwargs): """Initialize a transformer""" def __call__(self, text: str) -> str: """Apply the transformation""" ``` -------------------------------- ### Create and Apply Turkish Normalizer Pipeline Source: https://github.com/ysdede/trnorm/blob/master/README_SIMPLIFIED_NORMALIZER.md Illustrates the creation of a reusable normalization pipeline using custom sequences of transformers. This allows for efficient application of the same normalization steps to multiple text inputs. ```python from trnorm import create_normalizer_pipeline # Create a pipeline with custom normalizers pipeline = create_normalizer_pipeline([ "preprocess_dimensions", "normalize_dimensions", "convert_numbers", "normalize_units", "lowercase" ]) # Apply the pipeline to multiple texts texts = [ "Odanın boyutları 2x3x4 metre.", "Halının boyutu 120x180cm." ] for text in texts: normalized = pipeline.apply(text) print(normalized) ``` -------------------------------- ### Create Benchmark Transformer Pipeline Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Demonstrates creating a custom TransformerPipeline for benchmarking purposes. This pipeline includes specific transformers like 'preprocess_dimensions', 'normalize_dimensions', 'convert_numbers', etc., in a defined order. ```python from trnorm import TransformerPipeline # Create a pipeline for your benchmarking needs benchmark_pipeline = TransformerPipeline([ "preprocess_dimensions", "normalize_dimensions", "convert_numbers", "normalize_ordinals", "normalize_units", "lowercase" ]) # Process your benchmark data benchmark_results = [] for reference, hypothesis in benchmark_data: normalized_reference = benchmark_pipeline.apply(reference) normalized_hypothesis = benchmark_pipeline.apply(hypothesis) # Calculate metrics using normalized texts # ... benchmark_results.append(result) ``` -------------------------------- ### Handle Apostrophes in Turkish Suffixes (Python) Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md This code example illustrates the use of the `trnorm` library's `apply_apostrophe_handling` parameter to remove apostrophes used in Turkish suffixes, particularly with numbers and proper nouns. This results in more natural text by merging the suffix with the base word, such as '8'i' becoming 'sekizi'. ```python from trnorm import normalize # Enable apostrophe handling text = "8'i almadım" normalized_text = normalize(text, apply_apostrophe_handling=True) print(normalized_text) # Output: "sekizi almadım" # More examples text = "İstanbul'da yaşıyorum" normalized_text = normalize(text, apply_apostrophe_handling=True) print(normalized_text) # Output: "istanbulda yaşıyorum" ``` -------------------------------- ### Run trnorm Project Tests Source: https://github.com/ysdede/trnorm/blob/master/README.md This command executes the test suite for the trnorm project using Python's built-in unittest module. It discovers and runs all tests found within the project's test directory. This is a standard way to ensure the library's components are functioning as expected. ```bash python -m unittest discover ``` -------------------------------- ### Execute Ordinals Normalization Demo Source: https://github.com/ysdede/trnorm/blob/master/docs/README_ordinals.md This command runs the demo script for the Turkish Ordinals Normalization module. The demo script provides a practical showcase of the module's capabilities, illustrating how it converts various Turkish ordinal formats to their textual equivalents in a live execution. ```bash python demo_ordinals.py ``` -------------------------------- ### Custom TurkishNormalizer Class Initialization and Usage Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md Illustrates how to create and use a custom `TurkishNormalizer` instance by selectively enabling or disabling specific normalization features. This allows for fine-grained control over the normalization process. ```python from trnorm import TurkishNormalizer # Create a custom normalizer normalizer = TurkishNormalizer( apply_number_conversion=True, apply_ordinal_normalization=True, apply_symbol_conversion=True, apply_multiplication_symbol=True, apply_unit_normalization=True, apply_time_normalization=True, apply_legacy_normalization=False, lowercase=True, remove_hats=True ) # Normalize text text = "Ürün fiyatı 1.250,75 TL'dir." normalized_text = normalizer.normalize(text) print(normalized_text) # Output: "ürün fiyatı bin iki yüz elli virgül yetmiş beş türk lirası'dir." ``` -------------------------------- ### Apply Default Transformer Pipeline Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Shows how to use the default TransformerPipeline provided by the trnorm library. This pipeline applies a recommended set of transformers in an optimal order for Turkish text normalization. ```python from trnorm import TransformerPipeline pipeline = TransformerPipeline() normalized_text = pipeline.apply("Bugün 15. kattaki 3 toplantıya katıldım.") ``` -------------------------------- ### Creating and Registering Custom Transformers Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Shows how to define a new transformer function and register it with the TransformerPipeline. ```APIDOC ## Creating and Registering Custom Transformers ### Description This section explains how to create your own custom normalization transformer function and register it to be used within the `TransformerPipeline`. ### Method Function definition, helper function usage, and registration. ### Endpoint N/A (This is a library usage example, not a REST endpoint) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from trnorm import create_custom_transformer, register_transformer, TransformerPipeline def my_custom_transformer(text): # Your custom transformation logic here transformed_text = text.replace("example", "sample") return transformed_text custom_transformer = create_custom_transformer( name='my_transformer', func=my_custom_transformer, description='Replaces "example" with "sample"' ) register_transformer(custom_transformer) # Now you can use 'my_transformer' in a pipeline my_pipeline = TransformerPipeline(['my_transformer']) result = my_pipeline.apply("This is an example text.") print(result) ``` ### Response #### Success Response (200) Allows the use of the custom transformer in subsequent pipeline operations. #### Response Example ``` This is an sample text. ``` ``` -------------------------------- ### Migrate Old Turkish Normalizer API to Transformer Approach Source: https://github.com/ysdede/trnorm/blob/master/README_SIMPLIFIED_NORMALIZER.md Provides guidance on migrating from the legacy 'trnorm' API to the new transformer-based approach. It maps old optional parameters to their corresponding transformer names for seamless transition. ```python # Old API usage # normalize(text, apply_legacy_normalization=True, apply_apostrophe_handling=True) # New API usage with transformers normalize(text, transformers=["legacy_normalize"]) ``` -------------------------------- ### TransformerPipeline Usage Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Demonstrates how to use the TransformerPipeline with the default set of transformers. ```APIDOC ## TransformerPipeline Usage ### Description This example shows how to initialize and use the `TransformerPipeline` with its default configuration to normalize Turkish text. ### Method Instantiation and method call. ### Endpoint N/A (This is a library usage example, not a REST endpoint) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from trnorm import TransformerPipeline pipeline = TransformerPipeline() normalized_text = pipeline.apply("Bugün 15. kattaki 3 toplantıya katıldım.") print(normalized_text) ``` ### Response #### Success Response (200) Returns the normalized string. #### Response Example ``` bugün on beşinci kattaki üç toplantıya katıldım. ``` ``` -------------------------------- ### Basic String and List Normalization with `trnorm` Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md Demonstrates the fundamental usage of the `normalize` function for processing single strings and lists of strings in Turkish text normalization. It showcases automatic application of all default normalization rules. ```python from trnorm import normalize # Normalize a single string text = "Bugün 15. kattaki 3 toplantıya katıldım." normalized_text = normalize(text) print(normalized_text) # Output: "bugün on beşinci kattaki üç toplantıya katıldım." # Normalize a list of strings texts = [ "Saat 14:30'da %25 indirimli ürünler satışa çıkacak.", "II. Dünya Savaşı 1939-1945 yılları arasında gerçekleşti." ] normalized_texts = normalize(texts) print(normalized_texts) # Output: [ # "saat on dört otuzda yüzde yirmi beş indirimli ürünler satışa çıkacak.", # "ikinci dünya savaşı bin dokuz yüz otuz dokuz bin dokuz yüz kırk beş yılları arasında gerçekleşti." # ] ``` -------------------------------- ### Python: Using TransformerPipeline Class for Custom Normalization Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Illustrates creating and applying a custom normalization pipeline using the `TransformerPipeline` class. This provides an object-oriented way to manage and apply a sequence of transformations to Turkish text. ```python from trnorm import TransformerPipeline # Create a custom transformer pipeline pipeline = TransformerPipeline([ "preprocess_dimensions", "normalize_dimensions", "convert_numbers", "normalize_units", "lowercase" ]) # Apply the pipeline to text text = "Halının boyutu 120x180cm." normalized_text = pipeline.apply(text) print(normalized_text) # Output: "halının boyutu yüz yirmi çarpı yüz seksen santimetre." ``` -------------------------------- ### Custom TransformerPipeline Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Illustrates how to create a custom TransformerPipeline with a specific order of transformers. ```APIDOC ## Custom TransformerPipeline ### Description This example demonstrates creating a `TransformerPipeline` with a user-defined list of transformers, allowing for specific normalization sequences. ### Method Instantiation with custom transformers. ### Endpoint N/A (This is a library usage example, not a REST endpoint) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from trnorm import TransformerPipeline custom_pipeline = TransformerPipeline([ 'convert_numbers', 'normalize_ordinals', 'lowercase' ]) text_to_normalize = "Bugün 15. kattaki 3 toplantıya katıldım." normalized_text = custom_pipeline.apply(text_to_normalize) print(normalized_text) ``` ### Response #### Success Response (200) Returns the normalized string based on the custom pipeline. #### Response Example ``` bugün 15. kattaki 3 toplantıya katıldım. ``` ``` -------------------------------- ### Basic Turkish Text Normalization Source: https://github.com/ysdede/trnorm/blob/master/README_SIMPLIFIED_NORMALIZER.md Demonstrates the fundamental usage of the `normalize` function for Turkish text. It showcases default normalization and how to apply custom transformers for specific normalization tasks. ```python from trnorm import normalize # Use default normalizers normalized_text = normalize("Bugün 15. kattaki 3 toplantıya katıldım.") # Result: "bugün on beşinci kattaki üç toplantıya katıldım." # Use custom normalizers normalized_text = normalize("Dün 3x4 metre halı aldım.", transformers=["normalize_dimensions", "convert_numbers"]) # Result: "Dün üç çarpı dört metre halı aldım." ``` -------------------------------- ### Batch Processing with Lists - Python Source: https://context7.com/ysdede/trnorm/llms.txt Demonstrates how to process a list of Turkish text strings for normalization using the trnorm library. It shows both default and custom normalization pipelines, handling various cases like numbers, percentages, and currency. ```python from trnorm import normalize from trnorm.num_to_text import convert_numbers_to_words_wrapper from trnorm.text_utils import turkish_lower # Using the default pipeline (complete normalization) text = "Bugün 15 Nisan 2025'te %20 indirimle 1.500€ ödedim." normalized = normalize(text) print(normalized) # Output: "bugün on beş nisan iki bin yirmi beşte yüzde yirmi indirimle bin beş yüz avro ödedim" # Batch processing with lists texts = [ "1. gün 100 kişi geldi.", "Fiyat %50 indirimli, 250€.", "Saat 14:30'da toplantı var." ] normalized_texts = normalize(texts) for original, normalized in zip(texts, normalized_texts): print(f"Original: {original}") print(f"Normalized: {normalized}") print() # Output: # Original: 1. gün 100 kişi geldi. # Normalized: birinci gün yüz kişi geldi # # Original: Fiyat %50 indirimli, 250€. # Normalized: fiyat yüzde elli indirimli iki yüz elli avro # # Original: Saat 14:30'da toplantı var. # Normalized: saat on dört otuzda toplantı var # Custom normalization pipeline (selective transformations) custom_pipeline = [convert_numbers_to_words_wrapper, turkish_lower] text = "Bugün 25 kişi geldi ve 100 lira ödendi." normalized = normalize(text, converters=custom_pipeline) print(normalized) # Output: "bugün yirmi beş kişi geldi ve yüz lira ödendi." # Context-aware normalization for reference-hypothesis pairs reference = "Ankara ile İstanbul arası 450 km." hypothesis = "Ankara ile İstanbul arası 450 kilometre." # Normalize both with context awareness normalized_ref = normalize(reference) normalized_hyp = normalize(hypothesis, context_text=reference) print(f"Reference: {normalized_ref}") print(f"Hypothesis: {normalized_hyp}") # Output: # Reference: ankarayla istanbul arası dört yüz elli kilometre # Hypothesis: ankarayla istanbul arası dört yüz elli kilometre ``` -------------------------------- ### Run Ordinals Normalization Tests Source: https://github.com/ysdede/trnorm/blob/master/docs/README_ordinals.md This command executes the test suite for the Turkish Ordinals Normalization module using Python's built-in unittest framework. It's essential for verifying the module's functionality and ensuring all features, including edge cases and different ordinal formats, are handled correctly. ```bash python -m unittest test_ordinals.py ``` -------------------------------- ### Handling Dimensions and Multiplication Symbols Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md Demonstrates the library's capability to intelligently handle multiplication symbols within dimensions, converting expressions like '2x3x4' into their text equivalents. It also shows handling dimensions with units. ```python from trnorm import normalize # Handle dimensions with merged multiplication symbols text = "Odanın boyutları 2x3x4 metre." normalized_text = normalize(text) print(normalized_text) # Output: "odanın boyutları iki çarpı üç çarpı dört metre." # Handle dimensions with units text = "Halının boyutu 120x180cm." normalized_text = normalize(text) print(normalized_text) # Output: "halının boyutu yüz yirmi çarpı yüz seksen santimetre." ``` -------------------------------- ### Python: Extend Symbol Mappings at Runtime in trnorm Source: https://github.com/ysdede/trnorm/blob/master/docs/README_symbols.md Demonstrates extending the symbol mappings within the `trnorm` library at runtime. This involves using the `add_symbol_mapping` function to introduce new symbol-to-text conversions, such as mapping '§' to 'paragraf'. ```python from trnorm import add_symbol_mapping # Add a new symbol mapping add_symbol_mapping("§", "paragraf", False) # § symbol -> "paragraf" (before number) ``` -------------------------------- ### Python: Transform Turkish Text with Default Pipeline Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Demonstrates basic usage of the `transform` function from the `trnorm` library to normalize single strings and lists of strings using the default transformer pipeline. This is useful for common Turkish text normalization tasks. ```python from trnorm import transform # Normalize a single string using the default pipeline text = "Bugün 15. kattaki 3 toplantıya katıldım." normalized_text = transform(text) print(normalized_text) # Output: "bugün on beşinci kattaki üç toplantıya katıldım." # Normalize a list of strings texts = [ "Saat 14:30'da %25 indirimli ürünler satışa çıkacak.", "II. Dünya Savaşı 1939-1945 yılları arasında gerçekleşti." ] normalized_texts = transform(texts) print(normalized_texts) # Output: [ # "saat on dört otuzda yüzde yirmi beş indirimli ürünler satışa çıkacak.", # "ikinci dünya savaşı bin dokuz yüz otuz dokuz bin dokuz yüz kırk beş yılları arasında gerçekleşti." # ] ``` -------------------------------- ### Normalization of Time Expressions Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md Demonstrates how the `trnorm` library handles various time formats, converting expressions like 'Saat 22.00' or '13.30' into their natural language equivalents. It covers full times and standalone times. ```python from trnorm import normalize # Normalize time expressions text = "Saat 22.00'de toplantımız var." normalized_text = normalize(text) print(normalized_text) # Output: "saat yirmi ikide toplantımız var." # Normalize standalone times text = "13.30'da görüşeceğiz." normalized_text = normalize(text) print(normalized_text) # Output: "on üç buçukta görüşeceğiz." # Normalize times with minutes text = "Saat 10.15'te görüşelim." normalized_text = normalize(text) print(normalized_text) # Output: "saat on on beşte görüşelim." ``` -------------------------------- ### TransformerPipeline Class API Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Provides the API signature for the TransformerPipeline class, detailing its constructor and apply method. ```APIDOC ## TransformerPipeline Class API ### Description Reference documentation for the `TransformerPipeline` class, outlining its initialization parameters and the signature of its primary `apply` method. ### Method Class instantiation and method definition. ### Endpoint N/A (This is a library API reference, not a REST endpoint) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from typing import List, Union, Optional class TransformerPipeline: def __init__(self, transformers: Optional[List[str]] = None): """Initialize with list of transformer names""" pass def apply(self, text: Union[str, List[str]]) -> Union[str, List[str]]: """Apply all transformers in sequence""" pass ``` ### Response #### Success Response (200) N/A (This is code definition, not a response to a request) #### Response Example N/A ``` -------------------------------- ### Python: Transform Turkish Text with Custom Pipeline Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Shows how to use the `transform` function with a custom list of transformers to apply a specific normalization sequence. This allows for tailored normalization based on specific needs, overriding the default pipeline. ```python from trnorm import transform # Use a custom transformer pipeline text = "Odanın boyutları 2x3x4 metre." custom_pipeline = ["preprocess_dimensions", "normalize_dimensions", "convert_numbers"] normalized_text = transform(text, transformers=custom_pipeline) print(normalized_text) # Output: "Odanın boyutları iki çarpı üç çarpı dört metre." ``` -------------------------------- ### Python: Create Custom Converter Sets Source: https://github.com/ysdede/trnorm/blob/master/README_BASIC_NORMALIZER.md Demonstrates how to create and use different sets of conversion functions tailored for specific normalization tasks. This allows for flexible application of normalization rules, such as focusing only on numbers and symbols or performing full text normalization. ```python from trnorm import normalize from trnorm.num_to_text import convert_numbers_to_words_wrapper from trnorm.symbols import convert_symbols from trnorm.legacy_normalizer import normalize_text, replace_hatted_characters, turkish_lower from trnorm.dimension_utils import preprocess_dimensions, normalize_dimensions from trnorm.ordinals import normalize_ordinals from trnorm.unit_utils import normalize_units # Number and symbol conversion only number_symbol_converters = [ convert_symbols, convert_numbers_to_words_wrapper ] # Full text normalization full_normalization_converters = [ preprocess_dimensions, convert_symbols, normalize_dimensions, convert_numbers_to_words_wrapper, normalize_ordinals, normalize_units, replace_hatted_characters, turkish_lower ] # Legacy normalization (aggressive) legacy_converters = [ normalize_text ] # Apply different converter sets to the same text text = "Ürün boyutları 120x180cm ve fiyatı 1.250,75 TL'dir." number_symbol_result = normalize(text, number_symbol_converters) full_result = normalize(text, full_normalization_converters) legacy_result = normalize(text, legacy_converters) ``` -------------------------------- ### Python Turkish Text Case Conversion and Accent Removal Source: https://github.com/ysdede/trnorm/blob/master/docs/README_text_utils.md Demonstrates the usage of functions for converting Turkish text to lowercase, uppercase, and capitalized forms, as well as removing accents. Requires the 'text_utils' module. ```python from text_utils import turkish_lower, turkish_upper, turkish_capitalize, sapkasiz # Case conversion print(turkish_lower("İSTANBUL")) # "istanbul" print(turkish_upper("istanbul")) # "İSTANBUL" print(turkish_capitalize("istanbul")) # "İstanbul" # Accent removal print(sapkasiz("kâğıt")) # "kağıt" ``` -------------------------------- ### Python: Normalize Text with Multiple Functions Source: https://github.com/ysdede/trnorm/blob/master/README_BASIC_NORMALIZER.md Illustrates applying multiple conversion functions in sequence to a given text. The `normalize` function accepts a list of functions, executing them in the order they appear. ```python from trnorm import normalize from trnorm.num_to_text import convert_numbers_to_words_wrapper from trnorm.legacy_normalizer import turkish_lower # Apply multiple conversion functions in sequence normalized_text = normalize("Bugün 15 kişi geldi.", [ convert_numbers_to_words_wrapper, turkish_lower ]) # Result: "bugün on beş kişi geldi." ``` -------------------------------- ### Python: Import Available Conversion Functions Source: https://github.com/ysdede/trnorm/blob/master/README_BASIC_NORMALIZER.md Shows the various conversion functions that can be imported from the `trnorm` package for use in text normalization tasks. These include functions for number-to-text conversion, ordinals, symbols, and legacy normalization. ```python from trnorm.num_to_text import convert_numbers_to_words_wrapper from trnorm.ordinals import normalize_ordinals from trnorm.symbols import convert_symbols from trnorm.legacy_normalizer import normalize_text, replace_hatted_characters, turkish_lower from trnorm.dimension_utils import preprocess_dimensions, normalize_dimensions from trnorm.unit_utils import normalize_units ``` -------------------------------- ### Legacy Normalizer - Python Source: https://context7.com/ysdede/trnorm/llms.txt Provides a backward-compatible normalizer for basic Turkish text processing, focusing on lowercasing, removing hatted characters, and punctuation. It also demonstrates how to use the hatted character removal function independently. ```python from trnorm import normalize_text, replace_hatted_characters # Basic normalization (lowercase, remove hats, remove punctuation) text = "âîôû Çok iyi ve nazik biriydi. Prusya'daki ilk karşılaşmamızda onu konuşturmayı başarmıştım." normalized = normalize_text(text) print(normalized) # Output: "aiou çok iyi ve nazik biriydi prusyadaki ilk karşılaşmamızda onu konuşturmayı başarmıştım" # Only replace hatted characters (keep case and punctuation) text_with_hats = "âîôû Çok iyi ve Hukûk kitabı." cleaned = replace_hatted_characters(text_with_hats) print(cleaned) # Output: "aiou Çok iyi ve Hukuk kitabı." # Process a list of texts texts = [ "Turner'ın 'Köle Gemisi' isimli tablosuna bakıyoruz.", "Turner'ın Köle Gemisi isimli tablosuna bakıyoruz.", "Hukûk âlimi Profesör Âdem'in çalışmaları." ] normalized_texts = normalize_text(texts) for original, normalized in zip(texts, normalized_texts): print(f"Original: {original}") print(f"Normalized: {normalized}") print() # Output: # Original: Turner'ın 'Köle Gemisi' isimli tablosuna bakıyoruz. # Normalized: turnerın köle gemisi isimli tablosuna bakıyoruz # # Original: Turner'ın Köle Gemisi isimli tablosuna bakıyoruz. # Normalized: turnerın köle gemisi isimli tablosuna bakıyoruz # # Original: Hukûk âlimi Profesör Âdem'in çalışmaları. # Normalized: hukuk alimi profesör ademin çalışmaları # Compare legacy and modern normalizers text = "Bugün 25 Nisan'da %20 indirim var." legacy = normalize_text(text) modern = normalize(text) print(f"Legacy: {legacy}") print(f"Modern: {modern}") # Output: # Legacy: bugün 25 nisanda 20 indirim var # Modern: bugün yirmi beş nisanda yüzde yirmi indirim var ``` -------------------------------- ### Python: Create and Register Custom Turkish Text Transformers Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Details the process of defining a custom transformer function, creating a transformer object using `create_custom_transformer`, registering it with `register_transformer`, and then using it within a `TransformerPipeline`. This enables extending the library's functionality. ```python from trnorm import create_custom_transformer, register_transformer, TransformerPipeline # Define a custom transformer function def replace_turkish_month_names(text): """Replace Turkish month names with their numeric representation.""" month_mapping = { "ocak": "1", "şubat": "2", # ... other months } result = text for month, number in month_mapping.items(): result = result.replace(month, number) result = result.replace(month.capitalize(), number) return result # Create and register the custom transformer month_transformer = create_custom_transformer( name="replace_month_names", func=replace_turkish_month_names, description="Replace Turkish month names with their numeric representation" ) register_transformer(month_transformer) # Create a pipeline with the custom transformer custom_pipeline = TransformerPipeline([ "replace_month_names", "convert_numbers", "lowercase" ]) # Apply the pipeline text = "5 Ocak 2023 tarihinde toplantı var." normalized = custom_pipeline.apply(text) print(normalized) # Output: "beş bir iki bin yirmi üç tarihinde toplantı var." ``` -------------------------------- ### Python: Convert Percent and Currency Symbols using trnorm Source: https://github.com/ysdede/trnorm/blob/master/docs/README_symbols.md Demonstrates the basic usage of the `convert_symbols` function from the `trnorm` package to normalize percent and currency symbols in Turkish text. It handles symbols before and after numbers, including cases with apostrophes. ```python from trnorm import convert_symbols # Convert percent symbols text = "Rezerv oranım gene %10. Yani rezerv olarak 90 altın ayıracağım." normalized = convert_symbols(text) # Result: "Rezerv oranım gene yüzde 10. Yani rezerv olarak 90 altın ayıracağım." # Convert currency symbols (before numbers) text = "$50 değerindeki ürün" normalized = convert_symbols(text) # Result: "50 dolar değerindeki ürün" # Convert currency symbols (after numbers) text = "Fiyat: 75,5 €" normalized = convert_symbols(text) # Result: "Fiyat: 75,5 avro" # Convert mixed symbols text = "%10 indirimli ürün $50'ye satılıyor." normalized = convert_symbols(text) # Result: "yüzde 10 indirimli ürün 50 dolar'ye satılıyor." ``` -------------------------------- ### Python: Create Custom SymbolConverter in trnorm Source: https://github.com/ysdede/trnorm/blob/master/docs/README_symbols.md Illustrates how to create a custom `SymbolConverter` instance from the `trnorm` package for advanced symbol conversion needs. This method allows disabling default mappings and adding custom ones programmatically. ```python from trnorm import SymbolConverter # Create a custom converter without loading default mappings converter = SymbolConverter(load_defaults=False) # Add custom mappings converter.add_symbol_mapping("@", "et", False) converter.add_symbol_mapping("#", "numara", False) # Convert text using the custom converter text = "@50 ve #20" normalized = converter.convert_all_symbols(text) # Result: "et 50 ve numara 20" ``` -------------------------------- ### Create and Register Custom Transformer Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md Illustrates how to define and register a custom transformer within the trnorm framework. This involves creating a Python function for the transformation logic and then using `create_custom_transformer` and `register_transformer`. ```python from trnorm import create_custom_transformer, register_transformer def my_custom_transformer(text): # Custom transformation logic return transformed_text custom_transformer = create_custom_transformer( name='my_transformer', func=my_custom_transformer, description='My custom text transformation' ) register_transformer(custom_transformer) ```