TRNorm

https://github.com/ysdede/trnorm
Admin
TRNorm is a specialized Python package for normalizing Turkish text in Automatic Speech Recognition...
Tokens:21,763
Snippets:115
Trust Score:8.2
License:Apache-2.0
Update:11 months ago
Show doc for...
Tokens:
Raw
### Python Full Normalization Example

Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb

Demonstrates full text normalization using a predefined list of converters including dimension preprocessing, unit normalization, number to word conversion, symbol conversion, ordinal normalization, and Turkish lowercasing. It iterates through example texts, applies the normalization, and prints the original and normalized versions.

```python
print("\nFull normalization example:")
full_converters = [
    preprocess_dimensions,
    normalize_dimensions,
    normalize_units,
    convert_numbers_to_words_wrapper,
    convert_symbols,
    normalize_ordinals,
    turkish_lower
]

for text in example_texts[3:]:
    print(f"\nOriginal: {text}")
    normalized = normalize(text, full_converters)
    print(f"Normalized: {normalized}")
```

--------------------------------

### Get Available Turkish Normalizers

Source: https://github.com/ysdede/trnorm/blob/master/README_SIMPLIFIED_NORMALIZER.md

Shows how to retrieve a list of all available normalization functions within the 'trnorm' package. Each normalizer is presented with a brief explanation of its purpose.

```python
from trnorm import get_available_normalizers

normalizers = get_available_normalizers()
for name, description in normalizers.items():
    print(f"{name}: {description}")
```

--------------------------------

### Selective Normalization Steps with `normalize` Function

Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md

Shows how to apply only specific normalization steps by passing boolean arguments to the `normalize` function. This example demonstrates converting numbers to text while preserving case and diacritics, and also applying legacy normalization.

```python
from trnorm import normalize

# Only convert numbers to text, keep case and diacritical marks
text = "Âlim insanlar 15 kitap okumuş."
normalized_text = normalize(
    text,
    apply_number_conversion=True,
    apply_ordinal_normalization=False,
    apply_symbol_conversion=False,
    apply_multiplication_symbol=False,
    apply_unit_normalization=False,
    apply_time_normalization=False,
    lowercase=False,
    remove_hats=False
)
print(normalized_text)
# Output: "Âlim insanlar on beş kitap okumuş."

# Apply legacy normalization (more aggressive, removes punctuation)
text = "Bugün 3x4 metre halı aldım."
normalized_text = normalize(text, apply_legacy_normalization=True)
print(normalized_text)
# Output: "bugün üç çarpı dört metre halı aldım"
```

--------------------------------

### Basic Text Similarity Metrics Example - Python

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_metrics.md

Demonstrates the basic usage of WER, CER, and Levenshtein distance for evaluating similarity between two Turkish text strings. This example shows how to import and apply these metrics to assess differences in text.

```python
from trnorm.metrics import wer, cer, levenshtein_distance

reference = "Türkçe metin normalizasyonu önemlidir"
hypothesis = "Türkçe metin normalizasyon önemli"

# Calculate metrics
wer_score = wer(reference, hypothesis)
cer_score = cer(reference, hypothesis)
lev_distance = levenshtein_distance(reference, hypothesis)

print(f"WER: {wer_score:.4f}")
print(f"CER: {cer_score:.4f}")
print(f"Levenshtein Distance: {lev_distance}")
```

--------------------------------

### Python Custom Normalization Combinations

Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb

Shows how to apply custom combinations of converters for specific normalization tasks. Examples include normalizing only dimensions, only numbers to words, only units, and a full normalization.

```python
print("\nCustom converter combinations:")
text = "Ürün boyutları 120x180cm ve fiyatı 1.250,75 TL'dir."
print(f"\nOriginal: {text}")

# Only dimension handling
dimension_only = normalize(text, [preprocess_dimensions, normalize_dimensions])
print(f"Dimension only: {dimension_only}")

# Only number conversion
number_only = normalize(text, [convert_numbers_to_words_wrapper])
print(f"Number only: {number_only}")

# Only unit normalization
unit_only = normalize(text, [normalize_units])
print(f"Unit only: {unit_only}")

# Full normalization
full_norm = normalize(text, full_converters)
print(f"Full normalization: {full_norm}")
```

--------------------------------

### Demonstrate trnorm Normalization Functions in Python

Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb

This Python code showcases the usage of the `normalize` function from the trnorm library with different sets of converters. It demonstrates basic normalization including number-to-word conversion and lowercasing, as well as specialized normalization for dimensions and units. The code iterates through example texts, applies the specified converters, and prints both the original and normalized versions.

```python
# Example texts with various normalization needs
example_texts = [
    "Bugün 15. kattaki 3 toplantıya katıldım.",
    "Saat 14:30'da %25 indirimli ürünler satışa çıkacak.",
    "II. Dünya Savaşı 1939-1945 yılları arasında gerçekleşti.",
    "Ürün fiyatı 1.250,75 TL'dir.",
    "1. sınıfta 23 öğrenci var.",
    "Dün 3x4 metre halı aldım.",
    "Âlim insanlar bilgilerini paylaşır.",
]

# Examples for dimension and unit handling
mixed_examples = [
    "Odanın boyutları 2x3x4 metre.",
    "Halının boyutu 120x180cm.",
    "Ağırlığı 5 kg. ve uzunluğu 10 metre.",
    "Sıcaklık 25 °C ve nem %60.",
]

# Basic normalization example
print("Basic normalization example:")
basic_converters = [
    convert_numbers_to_words_wrapper,
    turkish_lower
]

for text in example_texts[:3]:
    print(f"\nOriginal: {text}")
    normalized = normalize(text, basic_converters)
    print(f"Normalized: {normalized}")

# Dimension and unit handling example
print("\nDimension and unit handling example:")
dimension_converters = [
    preprocess_dimensions,
    normalize_dimensions,
    normalize_units,
    convert_numbers_to_words_wrapper,
    turkish_lower
]

for text in mixed_examples[:2]:
    print(f"\nOriginal: {text}")
    normalized = normalize(text, dimension_converters)
    print(f"Normalized: {normalized}")

```

--------------------------------

### Python Batch Normalization Example

Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb

Illustrates batch processing of a list of texts using basic converters. The `normalize` function is applied to multiple texts simultaneously, and the results are iterated and printed with numbering.

```python
print("\nBatch processing example:")
batch_normalized = normalize(example_texts, basic_converters)
for i, normalized in enumerate(batch_normalized, 1):
    print(f"{i}. {normalized}")
```

--------------------------------

### Get Available Transformers and Descriptions

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Retrieves all available transformers and their descriptions from the trnorm library. It iterates through the returned dictionary and prints each transformer's name and its corresponding description.

```python
transformers = get_available_transformers()
for name, description in transformers.items():
    print(f"{name}: {description}")
```

--------------------------------

### Python: Get Available Transformers in trnorm

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Demonstrates how to retrieve a list of all available transformers that can be used within the `trnorm` library's pipelines. This function is useful for inspecting the library's capabilities.

```python
from trnorm import get_available_transformers


```

--------------------------------

### Unit Abbreviation Conversion to Full Text

Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md

Provides examples of converting common unit abbreviations (e.g., 'cm', 'm') into their full Turkish text forms ('santimetre', 'metre'). It also covers scenarios with multiple units and disabling unit normalization.

```python
from trnorm import normalize

# Convert unit abbreviations to full text
text = "Masanın yüksekliği 75 cm."
normalized_text = normalize(text)
print(normalized_text)
# Output: "masanın yüksekliği yetmiş beş santimetre."

# Multiple units in the same text
text = "Odanın boyutları 5 m x 4 m, yüksekliği 3 m."
normalized_text = normalize(text)
print(normalized_text)
# Output: "odanın boyutları beş metre çarpı dört metre, yüksekliği üç metre."

# Disable unit normalization
text = "Sıcaklık 25 °C."
normalized_text = normalize(text, apply_unit_normalization=False)
print(normalized_text)
# Output: "sıcaklık yirmi beş °c."
```

--------------------------------

### Apply Custom Normalization Pipeline in Python

Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb

This Python code defines a custom normalization pipeline using functions from the trnorm library. It then applies this pipeline to a list of example Turkish texts and prints the normalized output. The pipeline includes functions for time normalization, apostrophe removal, specific character handling (sapkasiz), dimension preprocessing, symbol conversion, and number-to-word conversion.

```python
# print(remove_apostrophes("Turner'ın 'Köle Gemisi' isimli tablosuna bakıyoruz."))
# print(remove_apostrophes("Turner'ın Köle Gemisi isimli tablosuna bakıyoruz."))
# print(remove_apostrophes("""Turner"ın "Köle Gemisi"isimli tablosuna bakıyoruz.""""))
# print(remove_apostrophes("""Turner"ın Köle Gemisi isimli tablosuna bakıyoruz.""""))
# print(remove_apostrophes("Turner'ın (Köle Gemisi) isimli tablosuna bakıyoruz."))
# print(remove_apostrophes("Turner'ın Köle Gemisi isimli tablosuna bakıyoruz."))
# print(remove_apostrophes('"Turner"ın "Köle Gemisi"isimli tablosuna bakıyoruz."'))
# print(remove_apostrophes('"'Turner"ın Köle Gemisi isimli tablosuna bakıyoruz.'"'))
# print()

# my_pipeline = [remove_apostrophes, normalize_ordinals, preprocess_dimensions, convert_numbers_to_words_wrapper]
my_pipeline = [normalize_times, remove_apostrophes, sapkasiz, preprocess_dimensions, convert_symbols, convert_numbers_to_words_wrapper]  # Changed order

def apply_normalizers(text):
    for norm in my_pipeline:
        text = norm(text)
    return text

example_texts = [
    "Turner'ın 'Köle Gemisi' isimli tablosuna bakıyoruz.",
    "Odanın boyutları 2x3x4 metre.",
    "Halının boyutu 120x180cm.",
    "Ağırlığı 5 kg. ve uzunluğu 10 metre.",
    "Sıcaklık 25 °C ve nem %60.",
    "Saat 14:30'da %25 indirimli ürünler satışa çıkacak.",
    "Ürün fiyatı 1.250,75 TL'dir.",
    "1. sınıfta 23 öğrenci var.",
    "Dün 3x4 metre halı aldım.",
    "II. Wilhelm, Almanya imparatoruydu.",
    "V. Karl, Kutsal Roma İmparatoru'ydu.",
    "1'inci sınıf 1inci kat",
    "2. Mahmud ya da II. Mahmud. BUG",
    "1.000 ₺",
    "$500",
    "600 $",
    "kâğıt ve kalem",
    "Âdem ile Havva",
    "îmanın kimde",
    "Îman büyük harfle şapkalı",
    "hûr ne",
    "Yavru ile kâtip"
    ]

for e in example_texts:
    print(apply_normalizers(e))
```

--------------------------------

### Integration of Metrics with Normalization - Python

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_metrics.md

Illustrates how the metrics module can be integrated with other trnorm modules, specifically `convert_numbers_to_words_wrapper`. This example shows how text normalization (converting numbers to words) can potentially improve WER scores when comparing ASR output against a reference.

```python
from trnorm.metrics import wer
from trnorm.num_to_text import convert_numbers_to_words_wrapper

# Original text with numbers
reference = "Bu 42 sayısı önemlidir"
# Text with normalized numbers
normalized = convert_numbers_to_words_wrapper(reference)
# ASR output
hypothesis = "bu kırk iki sayısı önemli"

# Calculate WER between original and ASR output
original_wer = wer(reference, hypothesis)
# Calculate WER between normalized and ASR output
normalized_wer = wer(normalized, hypothesis)

print(f"Original WER: {original_wer:.4f}")
print(f"Normalized WER: {normalized_wer:.4f}")
```

--------------------------------

### Process ASR Logs with WER and CER - Python

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_metrics.md

An example script snippet for processing Automatic Speech Recognition (ASR) log files. It iterates through log entries, extracts reference and hypothesis texts, and calculates WER and CER scores for each entry to evaluate ASR performance.

```python
from trnorm.metrics import wer, cer

# For each line in an ASR log file
for line in asr_log:
    reference = line["reference_text"]
    hypothesis = line["asr_output"]
    
    # Calculate metrics
    wer_score = wer(reference, hypothesis)
    cer_score = cer(reference, hypothesis)
    
    # Process or store results
    print(f"WER: {wer_score:.4f}, CER: {cer_score:.4f}")
```

--------------------------------

### Python: Create Custom Conversion Function

Source: https://github.com/ysdede/trnorm/blob/master/README_BASIC_NORMALIZER.md

Illustrates how to define and integrate a custom conversion function into the normalization process. The example shows a function to replace Turkish month names with their numeric representations, which can then be added to the list of converters used by `normalize`.

```python
from trnorm import normalize
from trnorm.num_to_text import convert_numbers_to_words_wrapper
from trnorm.legacy_normalizer import turkish_lower

def replace_turkish_month_names(text):
    """Replace Turkish month names with their numeric representation."""
    month_mapping = {
        "ocak": "1",
        "şubat": "2",
        # ... other months
    }
    
    result = text
    for month, number in month_mapping.items():
        result = result.replace(month, number)
        result = result.replace(month.capitalize(), number)
    
    return result

# Add your custom function to the converter list
custom_converters = [
    replace_turkish_month_names,
    convert_numbers_to_words_wrapper,
    turkish_lower
]

normalized_text = normalize("5 Ocak 2023 tarihinde toplantı var.", custom_converters)
# Result: "beş bir iki bin yirmi üç tarihinde toplantı var."
```

--------------------------------

### Python Unit Test Execution for Turkish Text Utilities

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_text_utils.md

Shows how to run the unit tests for the Turkish Text Utilities module using Python's unittest framework. This command executes tests defined in 'test_text_utils.py'.

```python
python -m unittest test_text_utils.py
```

--------------------------------

### TransformerPipeline Class Initialization and Apply Method

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Provides the API reference for the TransformerPipeline class, detailing its constructor which accepts an optional list of transformer names, and its 'apply' method for processing text.

```python
from typing import Optional, List, Union

class TransformerPipeline:
    def __init__(self, transformers: Optional[List[str]] = None):
        """Initialize with list of transformer names"""
        pass

    def apply(self, text: Union[str, List[str]]) -> Union[str, List[str]]:
        """Apply all transformers in sequence"""
        pass
```

--------------------------------

### Import trnorm Utilities in Python

Source: https://github.com/ysdede/trnorm/blob/master/examples/scratchpad.ipynb

This snippet demonstrates importing various normalization functions from the trnorm package. It includes utilities for dimensions, units, numbers to words, ordinals, symbols, apostrophe handling, and text manipulation.

```python
import sys
import os

# Add the parent directory to the path to import the trnorm package
# sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from trnorm import normalize
from trnorm.dimension_utils import preprocess_dimensions, normalize_dimensions
from trnorm.unit_utils import normalize_units
from trnorm.num_to_text import convert_numbers_to_words_wrapper
from trnorm.ordinals import normalize_ordinals
from trnorm.symbols import convert_symbols
from trnorm.apostrophe_handler import remove_apostrophes
from trnorm.text_utils import (
    turkish_lower,
    turkish_upper,
    turkish_capitalize,
    is_turkish_upper,
    sapkasiz
)
from trnorm.time_utils import normalize_times
from trnorm.symbols import SymbolConverter, convert_symbols, add_symbol_mapping
from trnorm.symbol_mappings import get_all_mappings
```

--------------------------------

### Initialize and Apply Transformer (Python)

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

This Python code defines a Transformer class. The `__init__` method initializes the transformer with a name, a transformation function, a description, and optional keyword arguments. The `__call__` method applies the defined transformation to an input string, returning the transformed string.

```python
class Transformer:
    def __init__(self, name: str, func: TransformerFunc, description: str, **kwargs):
        """Initialize a transformer"""

    def __call__(self, text: str) -> str:
        """Apply the transformation"""
```

--------------------------------

### Create and Apply Turkish Normalizer Pipeline

Source: https://github.com/ysdede/trnorm/blob/master/README_SIMPLIFIED_NORMALIZER.md

Illustrates the creation of a reusable normalization pipeline using custom sequences of transformers. This allows for efficient application of the same normalization steps to multiple text inputs.

```python
from trnorm import create_normalizer_pipeline

# Create a pipeline with custom normalizers
pipeline = create_normalizer_pipeline([
    "preprocess_dimensions",
    "normalize_dimensions",
    "convert_numbers",
    "normalize_units",
    "lowercase"
])

# Apply the pipeline to multiple texts
texts = [
    "Odanın boyutları 2x3x4 metre.",
    "Halının boyutu 120x180cm."
]

for text in texts:
    normalized = pipeline.apply(text)
    print(normalized)
```

--------------------------------

### Create Benchmark Transformer Pipeline

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Demonstrates creating a custom TransformerPipeline for benchmarking purposes. This pipeline includes specific transformers like 'preprocess_dimensions', 'normalize_dimensions', 'convert_numbers', etc., in a defined order.

```python
from trnorm import TransformerPipeline

# Create a pipeline for your benchmarking needs
benchmark_pipeline = TransformerPipeline([
    "preprocess_dimensions",
    "normalize_dimensions",
    "convert_numbers",
    "normalize_ordinals",
    "normalize_units",
    "lowercase"
])

# Process your benchmark data
benchmark_results = []
for reference, hypothesis in benchmark_data:
    normalized_reference = benchmark_pipeline.apply(reference)
    normalized_hypothesis = benchmark_pipeline.apply(hypothesis)
    # Calculate metrics using normalized texts
    # ...
    benchmark_results.append(result)
```

--------------------------------

### Handle Apostrophes in Turkish Suffixes (Python)

Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md

This code example illustrates the use of the `trnorm` library's `apply_apostrophe_handling` parameter to remove apostrophes used in Turkish suffixes, particularly with numbers and proper nouns. This results in more natural text by merging the suffix with the base word, such as '8'i' becoming 'sekizi'.

```python
from trnorm import normalize

# Enable apostrophe handling
text = "8'i almadım"
normalized_text = normalize(text, apply_apostrophe_handling=True)
print(normalized_text)
# Output: "sekizi almadım"

# More examples
text = "İstanbul'da yaşıyorum"
normalized_text = normalize(text, apply_apostrophe_handling=True)
print(normalized_text)
# Output: "istanbulda yaşıyorum"
```

--------------------------------

### Run trnorm Project Tests

Source: https://github.com/ysdede/trnorm/blob/master/README.md

This command executes the test suite for the trnorm project using Python's built-in unittest module. It discovers and runs all tests found within the project's test directory. This is a standard way to ensure the library's components are functioning as expected.

```bash
python -m unittest discover
```

--------------------------------

### Execute Ordinals Normalization Demo

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_ordinals.md

This command runs the demo script for the Turkish Ordinals Normalization module. The demo script provides a practical showcase of the module's capabilities, illustrating how it converts various Turkish ordinal formats to their textual equivalents in a live execution.

```bash
python demo_ordinals.py
```

--------------------------------

### Custom TurkishNormalizer Class Initialization and Usage

Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md

Illustrates how to create and use a custom `TurkishNormalizer` instance by selectively enabling or disabling specific normalization features. This allows for fine-grained control over the normalization process.

```python
from trnorm import TurkishNormalizer

# Create a custom normalizer
normalizer = TurkishNormalizer(
    apply_number_conversion=True,
    apply_ordinal_normalization=True,
    apply_symbol_conversion=True,
    apply_multiplication_symbol=True,
    apply_unit_normalization=True,
    apply_time_normalization=True,
    apply_legacy_normalization=False,
    lowercase=True,
    remove_hats=True
)

# Normalize text
text = "Ürün fiyatı 1.250,75 TL'dir."
normalized_text = normalizer.normalize(text)
print(normalized_text)
# Output: "ürün fiyatı bin iki yüz elli virgül yetmiş beş türk lirası'dir."
```

--------------------------------

### Apply Default Transformer Pipeline

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Shows how to use the default TransformerPipeline provided by the trnorm library. This pipeline applies a recommended set of transformers in an optimal order for Turkish text normalization.

```python
from trnorm import TransformerPipeline

pipeline = TransformerPipeline()
normalized_text = pipeline.apply("Bugün 15. kattaki 3 toplantıya katıldım.")
```

--------------------------------

### Creating and Registering Custom Transformers

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Shows how to define a new transformer function and register it with the TransformerPipeline.

```APIDOC
## Creating and Registering Custom Transformers

### Description
This section explains how to create your own custom normalization transformer function and register it to be used within the `TransformerPipeline`.

### Method
Function definition, helper function usage, and registration.

### Endpoint
N/A (This is a library usage example, not a REST endpoint)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from trnorm import create_custom_transformer, register_transformer, TransformerPipeline

def my_custom_transformer(text):
    # Your custom transformation logic here
    transformed_text = text.replace("example", "sample")
    return transformed_text

custom_transformer = create_custom_transformer(
    name='my_transformer',
    func=my_custom_transformer,
    description='Replaces "example" with "sample"'
)
register_transformer(custom_transformer)

# Now you can use 'my_transformer' in a pipeline
my_pipeline = TransformerPipeline(['my_transformer'])
result = my_pipeline.apply("This is an example text.")
print(result)
```

### Response
#### Success Response (200)
Allows the use of the custom transformer in subsequent pipeline operations.

#### Response Example
```
This is an sample text.
```
```

--------------------------------

### Migrate Old Turkish Normalizer API to Transformer Approach

Source: https://github.com/ysdede/trnorm/blob/master/README_SIMPLIFIED_NORMALIZER.md

Provides guidance on migrating from the legacy 'trnorm' API to the new transformer-based approach. It maps old optional parameters to their corresponding transformer names for seamless transition.

```python
# Old API usage
# normalize(text, apply_legacy_normalization=True, apply_apostrophe_handling=True)

# New API usage with transformers
normalize(text, transformers=["legacy_normalize"])
```

--------------------------------

### TransformerPipeline Usage

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Demonstrates how to use the TransformerPipeline with the default set of transformers.

```APIDOC
## TransformerPipeline Usage

### Description
This example shows how to initialize and use the `TransformerPipeline` with its default configuration to normalize Turkish text.

### Method
Instantiation and method call.

### Endpoint
N/A (This is a library usage example, not a REST endpoint)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from trnorm import TransformerPipeline

pipeline = TransformerPipeline()
normalized_text = pipeline.apply("Bugün 15. kattaki 3 toplantıya katıldım.")
print(normalized_text)
```

### Response
#### Success Response (200)
Returns the normalized string.

#### Response Example
```
bugün on beşinci kattaki üç toplantıya katıldım.
```
```

--------------------------------

### Basic String and List Normalization with `trnorm`

Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md

Demonstrates the fundamental usage of the `normalize` function for processing single strings and lists of strings in Turkish text normalization. It showcases automatic application of all default normalization rules.

```python
from trnorm import normalize

# Normalize a single string
text = "Bugün 15. kattaki 3 toplantıya katıldım."
normalized_text = normalize(text)
print(normalized_text)
# Output: "bugün on beşinci kattaki üç toplantıya katıldım."

# Normalize a list of strings
texts = [
    "Saat 14:30'da %25 indirimli ürünler satışa çıkacak.",
    "II. Dünya Savaşı 1939-1945 yılları arasında gerçekleşti."
]
normalized_texts = normalize(texts)
print(normalized_texts)
# Output: [
#   "saat on dört otuzda yüzde yirmi beş indirimli ürünler satışa çıkacak.",
#   "ikinci dünya savaşı bin dokuz yüz otuz dokuz bin dokuz yüz kırk beş yılları arasında gerçekleşti."
# ]
```

--------------------------------

### Python: Using TransformerPipeline Class for Custom Normalization

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Illustrates creating and applying a custom normalization pipeline using the `TransformerPipeline` class. This provides an object-oriented way to manage and apply a sequence of transformations to Turkish text.

```python
from trnorm import TransformerPipeline

# Create a custom transformer pipeline
pipeline = TransformerPipeline([
    "preprocess_dimensions",
    "normalize_dimensions",
    "convert_numbers",
    "normalize_units",
    "lowercase"
])

# Apply the pipeline to text
text = "Halının boyutu 120x180cm."
normalized_text = pipeline.apply(text)
print(normalized_text)
# Output: "halının boyutu yüz yirmi çarpı yüz seksen santimetre."
```

--------------------------------

### Custom TransformerPipeline

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Illustrates how to create a custom TransformerPipeline with a specific order of transformers.

```APIDOC
## Custom TransformerPipeline

### Description
This example demonstrates creating a `TransformerPipeline` with a user-defined list of transformers, allowing for specific normalization sequences.

### Method
Instantiation with custom transformers.

### Endpoint
N/A (This is a library usage example, not a REST endpoint)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from trnorm import TransformerPipeline

custom_pipeline = TransformerPipeline([
    'convert_numbers',
    'normalize_ordinals',
    'lowercase'
])

text_to_normalize = "Bugün 15. kattaki 3 toplantıya katıldım."
normalized_text = custom_pipeline.apply(text_to_normalize)
print(normalized_text)
```

### Response
#### Success Response (200)
Returns the normalized string based on the custom pipeline.

#### Response Example
```
bugün 15. kattaki 3 toplantıya katıldım.
```
```

--------------------------------

### Basic Turkish Text Normalization

Source: https://github.com/ysdede/trnorm/blob/master/README_SIMPLIFIED_NORMALIZER.md

Demonstrates the fundamental usage of the `normalize` function for Turkish text. It showcases default normalization and how to apply custom transformers for specific normalization tasks.

```python
from trnorm import normalize

# Use default normalizers
normalized_text = normalize("Bugün 15. kattaki 3 toplantıya katıldım.")
# Result: "bugün on beşinci kattaki üç toplantıya katıldım."

# Use custom normalizers
normalized_text = normalize("Dün 3x4 metre halı aldım.", 
                           transformers=["normalize_dimensions", "convert_numbers"])
# Result: "Dün üç çarpı dört metre halı aldım."
```

--------------------------------

### Batch Processing with Lists - Python

Source: https://context7.com/ysdede/trnorm/llms.txt

Demonstrates how to process a list of Turkish text strings for normalization using the trnorm library. It shows both default and custom normalization pipelines, handling various cases like numbers, percentages, and currency.

```python
from trnorm import normalize
from trnorm.num_to_text import convert_numbers_to_words_wrapper
from trnorm.text_utils import turkish_lower

# Using the default pipeline (complete normalization)
text = "Bugün 15 Nisan 2025'te %20 indirimle 1.500€ ödedim."
normalized = normalize(text)
print(normalized)
# Output: "bugün on beş nisan iki bin yirmi beşte yüzde yirmi indirimle bin beş yüz avro ödedim"

# Batch processing with lists
texts = [
    "1. gün 100 kişi geldi.",
    "Fiyat %50 indirimli, 250€.",
    "Saat 14:30'da toplantı var."
]
normalized_texts = normalize(texts)
for original, normalized in zip(texts, normalized_texts):
    print(f"Original: {original}")
    print(f"Normalized: {normalized}")
    print()
# Output:
# Original: 1. gün 100 kişi geldi.
# Normalized: birinci gün yüz kişi geldi
#
# Original: Fiyat %50 indirimli, 250€.
# Normalized: fiyat yüzde elli indirimli iki yüz elli avro
#
# Original: Saat 14:30'da toplantı var.
# Normalized: saat on dört otuzda toplantı var

# Custom normalization pipeline (selective transformations)
custom_pipeline = [convert_numbers_to_words_wrapper, turkish_lower]
text = "Bugün 25 kişi geldi ve 100 lira ödendi."
normalized = normalize(text, converters=custom_pipeline)
print(normalized)
# Output: "bugün yirmi beş kişi geldi ve yüz lira ödendi."

# Context-aware normalization for reference-hypothesis pairs
reference = "Ankara ile İstanbul arası 450 km."
hypothesis = "Ankara ile İstanbul arası 450 kilometre."
# Normalize both with context awareness
normalized_ref = normalize(reference)
normalized_hyp = normalize(hypothesis, context_text=reference)
print(f"Reference: {normalized_ref}")
print(f"Hypothesis: {normalized_hyp}")
# Output:
# Reference: ankarayla istanbul arası dört yüz elli kilometre
# Hypothesis: ankarayla istanbul arası dört yüz elli kilometre
```

--------------------------------

### Run Ordinals Normalization Tests

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_ordinals.md

This command executes the test suite for the Turkish Ordinals Normalization module using Python's built-in unittest framework. It's essential for verifying the module's functionality and ensuring all features, including edge cases and different ordinal formats, are handled correctly.

```bash
python -m unittest test_ordinals.py
```

--------------------------------

### Handling Dimensions and Multiplication Symbols

Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md

Demonstrates the library's capability to intelligently handle multiplication symbols within dimensions, converting expressions like '2x3x4' into their text equivalents. It also shows handling dimensions with units.

```python
from trnorm import normalize

# Handle dimensions with merged multiplication symbols
text = "Odanın boyutları 2x3x4 metre."
normalized_text = normalize(text)
print(normalized_text)
# Output: "odanın boyutları iki çarpı üç çarpı dört metre."

# Handle dimensions with units
text = "Halının boyutu 120x180cm."
normalized_text = normalize(text)
print(normalized_text)
# Output: "halının boyutu yüz yirmi çarpı yüz seksen santimetre."
```

--------------------------------

### Python: Extend Symbol Mappings at Runtime in trnorm

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_symbols.md

Demonstrates extending the symbol mappings within the `trnorm` library at runtime. This involves using the `add_symbol_mapping` function to introduce new symbol-to-text conversions, such as mapping '§' to 'paragraf'.

```python
from trnorm import add_symbol_mapping

# Add a new symbol mapping
add_symbol_mapping("§", "paragraf", False)  # § symbol -> "paragraf" (before number)
```

--------------------------------

### Python: Transform Turkish Text with Default Pipeline

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Demonstrates basic usage of the `transform` function from the `trnorm` library to normalize single strings and lists of strings using the default transformer pipeline. This is useful for common Turkish text normalization tasks.

```python
from trnorm import transform

# Normalize a single string using the default pipeline
text = "Bugün 15. kattaki 3 toplantıya katıldım."
normalized_text = transform(text)
print(normalized_text)
# Output: "bugün on beşinci kattaki üç toplantıya katıldım."

# Normalize a list of strings
texts = [
    "Saat 14:30'da %25 indirimli ürünler satışa çıkacak.",
    "II. Dünya Savaşı 1939-1945 yılları arasında gerçekleşti."
]
normalized_texts = transform(texts)
print(normalized_texts)
# Output: [
#   "saat on dört otuzda yüzde yirmi beş indirimli ürünler satışa çıkacak.",
#   "ikinci dünya savaşı bin dokuz yüz otuz dokuz bin dokuz yüz kırk beş yılları arasında gerçekleşti."
# ]
```

--------------------------------

### Normalization of Time Expressions

Source: https://github.com/ysdede/trnorm/blob/master/docs/normalizer.md

Demonstrates how the `trnorm` library handles various time formats, converting expressions like 'Saat 22.00' or '13.30' into their natural language equivalents. It covers full times and standalone times.

```python
from trnorm import normalize

# Normalize time expressions
text = "Saat 22.00'de toplantımız var."
normalized_text = normalize(text)
print(normalized_text)
# Output: "saat yirmi ikide toplantımız var."

# Normalize standalone times
text = "13.30'da görüşeceğiz."
normalized_text = normalize(text)
print(normalized_text)
# Output: "on üç buçukta görüşeceğiz."

# Normalize times with minutes
text = "Saat 10.15'te görüşelim."
normalized_text = normalize(text)
print(normalized_text)
# Output: "saat on on beşte görüşelim."
```

--------------------------------

### TransformerPipeline Class API

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Provides the API signature for the TransformerPipeline class, detailing its constructor and apply method.

```APIDOC
## TransformerPipeline Class API

### Description
Reference documentation for the `TransformerPipeline` class, outlining its initialization parameters and the signature of its primary `apply` method.

### Method
Class instantiation and method definition.

### Endpoint
N/A (This is a library API reference, not a REST endpoint)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from typing import List, Union, Optional

class TransformerPipeline:
    def __init__(self, transformers: Optional[List[str]] = None):
        """Initialize with list of transformer names"""
        pass

    def apply(self, text: Union[str, List[str]]) -> Union[str, List[str]]:
        """Apply all transformers in sequence"""
        pass
```

### Response
#### Success Response (200)
N/A (This is code definition, not a response to a request)

#### Response Example
N/A
```

--------------------------------

### Python: Transform Turkish Text with Custom Pipeline

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Shows how to use the `transform` function with a custom list of transformers to apply a specific normalization sequence. This allows for tailored normalization based on specific needs, overriding the default pipeline.

```python
from trnorm import transform

# Use a custom transformer pipeline
text = "Odanın boyutları 2x3x4 metre."
custom_pipeline = ["preprocess_dimensions", "normalize_dimensions", "convert_numbers"]
normalized_text = transform(text, transformers=custom_pipeline)
print(normalized_text)
# Output: "Odanın boyutları iki çarpı üç çarpı dört metre."
```

--------------------------------

### Python: Create Custom Converter Sets

Source: https://github.com/ysdede/trnorm/blob/master/README_BASIC_NORMALIZER.md

Demonstrates how to create and use different sets of conversion functions tailored for specific normalization tasks. This allows for flexible application of normalization rules, such as focusing only on numbers and symbols or performing full text normalization.

```python
from trnorm import normalize
from trnorm.num_to_text import convert_numbers_to_words_wrapper
from trnorm.symbols import convert_symbols
from trnorm.legacy_normalizer import normalize_text, replace_hatted_characters, turkish_lower
from trnorm.dimension_utils import preprocess_dimensions, normalize_dimensions
from trnorm.ordinals import normalize_ordinals
from trnorm.unit_utils import normalize_units

# Number and symbol conversion only
number_symbol_converters = [
    convert_symbols,
    convert_numbers_to_words_wrapper
]

# Full text normalization
full_normalization_converters = [
    preprocess_dimensions,
    convert_symbols,
    normalize_dimensions,
    convert_numbers_to_words_wrapper,
    normalize_ordinals,
    normalize_units,
    replace_hatted_characters,
    turkish_lower
]

# Legacy normalization (aggressive)
legacy_converters = [
    normalize_text
]

# Apply different converter sets to the same text
text = "Ürün boyutları 120x180cm ve fiyatı 1.250,75 TL'dir."

number_symbol_result = normalize(text, number_symbol_converters)
full_result = normalize(text, full_normalization_converters)
legacy_result = normalize(text, legacy_converters)
```

--------------------------------

### Python Turkish Text Case Conversion and Accent Removal

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_text_utils.md

Demonstrates the usage of functions for converting Turkish text to lowercase, uppercase, and capitalized forms, as well as removing accents. Requires the 'text_utils' module.

```python
from text_utils import turkish_lower, turkish_upper, turkish_capitalize, sapkasiz

# Case conversion
print(turkish_lower("İSTANBUL"))  # "istanbul"
print(turkish_upper("istanbul"))  # "İSTANBUL"
print(turkish_capitalize("istanbul"))  # "İstanbul"

# Accent removal
print(sapkasiz("kâğıt"))  # "kağıt"
```

--------------------------------

### Python: Normalize Text with Multiple Functions

Source: https://github.com/ysdede/trnorm/blob/master/README_BASIC_NORMALIZER.md

Illustrates applying multiple conversion functions in sequence to a given text. The `normalize` function accepts a list of functions, executing them in the order they appear.

```python
from trnorm import normalize
from trnorm.num_to_text import convert_numbers_to_words_wrapper
from trnorm.legacy_normalizer import turkish_lower

# Apply multiple conversion functions in sequence
normalized_text = normalize("Bugün 15 kişi geldi.", [
    convert_numbers_to_words_wrapper,
    turkish_lower
])
# Result: "bugün on beş kişi geldi."
```

--------------------------------

### Python: Import Available Conversion Functions

Source: https://github.com/ysdede/trnorm/blob/master/README_BASIC_NORMALIZER.md

Shows the various conversion functions that can be imported from the `trnorm` package for use in text normalization tasks. These include functions for number-to-text conversion, ordinals, symbols, and legacy normalization.

```python
from trnorm.num_to_text import convert_numbers_to_words_wrapper
from trnorm.ordinals import normalize_ordinals
from trnorm.symbols import convert_symbols
from trnorm.legacy_normalizer import normalize_text, replace_hatted_characters, turkish_lower
from trnorm.dimension_utils import preprocess_dimensions, normalize_dimensions
from trnorm.unit_utils import normalize_units
```

--------------------------------

### Legacy Normalizer - Python

Source: https://context7.com/ysdede/trnorm/llms.txt

Provides a backward-compatible normalizer for basic Turkish text processing, focusing on lowercasing, removing hatted characters, and punctuation. It also demonstrates how to use the hatted character removal function independently.

```python
from trnorm import normalize_text, replace_hatted_characters

# Basic normalization (lowercase, remove hats, remove punctuation)
text = "âîôû Çok iyi ve nazik biriydi. Prusya'daki ilk karşılaşmamızda onu konuşturmayı başarmıştım."
normalized = normalize_text(text)
print(normalized)
# Output: "aiou çok iyi ve nazik biriydi prusyadaki ilk karşılaşmamızda onu konuşturmayı başarmıştım"

# Only replace hatted characters (keep case and punctuation)
text_with_hats = "âîôû Çok iyi ve Hukûk kitabı."
cleaned = replace_hatted_characters(text_with_hats)
print(cleaned)
# Output: "aiou Çok iyi ve Hukuk kitabı."

# Process a list of texts
texts = [
    "Turner'ın 'Köle Gemisi' isimli tablosuna bakıyoruz.",
    "Turner'ın Köle Gemisi isimli tablosuna bakıyoruz.",
    "Hukûk âlimi Profesör Âdem'in çalışmaları."
]
normalized_texts = normalize_text(texts)
for original, normalized in zip(texts, normalized_texts):
    print(f"Original: {original}")
    print(f"Normalized: {normalized}")
    print()
# Output:
# Original: Turner'ın 'Köle Gemisi' isimli tablosuna bakıyoruz.
# Normalized: turnerın köle gemisi isimli tablosuna bakıyoruz
#
# Original: Turner'ın Köle Gemisi isimli tablosuna bakıyoruz.
# Normalized: turnerın köle gemisi isimli tablosuna bakıyoruz
#
# Original: Hukûk âlimi Profesör Âdem'in çalışmaları.
# Normalized: hukuk alimi profesör ademin çalışmaları

# Compare legacy and modern normalizers
text = "Bugün 25 Nisan'da %20 indirim var."
legacy = normalize_text(text)
modern = normalize(text)
print(f"Legacy: {legacy}")
print(f"Modern: {modern}")
# Output:
# Legacy: bugün 25 nisanda 20 indirim var
# Modern: bugün yirmi beş nisanda yüzde yirmi indirim var
```

--------------------------------

### Python: Create and Register Custom Turkish Text Transformers

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Details the process of defining a custom transformer function, creating a transformer object using `create_custom_transformer`, registering it with `register_transformer`, and then using it within a `TransformerPipeline`. This enables extending the library's functionality.

```python
from trnorm import create_custom_transformer, register_transformer, TransformerPipeline

# Define a custom transformer function
def replace_turkish_month_names(text):
    """Replace Turkish month names with their numeric representation."""
    month_mapping = {
        "ocak": "1",
        "şubat": "2",
        # ... other months
    }
    
    result = text
    for month, number in month_mapping.items():
        result = result.replace(month, number)
        result = result.replace(month.capitalize(), number)
    
    return result

# Create and register the custom transformer
month_transformer = create_custom_transformer(
    name="replace_month_names",
    func=replace_turkish_month_names,
    description="Replace Turkish month names with their numeric representation"
)
register_transformer(month_transformer)

# Create a pipeline with the custom transformer
custom_pipeline = TransformerPipeline([
    "replace_month_names",
    "convert_numbers",
    "lowercase"
])

# Apply the pipeline
text = "5 Ocak 2023 tarihinde toplantı var."
normalized = custom_pipeline.apply(text)
print(normalized)
# Output: "beş bir iki bin yirmi üç tarihinde toplantı var."
```

--------------------------------

### Python: Convert Percent and Currency Symbols using trnorm

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_symbols.md

Demonstrates the basic usage of the `convert_symbols` function from the `trnorm` package to normalize percent and currency symbols in Turkish text. It handles symbols before and after numbers, including cases with apostrophes.

```python
from trnorm import convert_symbols

# Convert percent symbols
text = "Rezerv oranım gene %10. Yani rezerv olarak 90 altın ayıracağım."
normalized = convert_symbols(text)
# Result: "Rezerv oranım gene yüzde 10. Yani rezerv olarak 90 altın ayıracağım."

# Convert currency symbols (before numbers)
text = "$50 değerindeki ürün"
normalized = convert_symbols(text)
# Result: "50 dolar değerindeki ürün"

# Convert currency symbols (after numbers)
text = "Fiyat: 75,5 €"
normalized = convert_symbols(text)
# Result: "Fiyat: 75,5 avro"

# Convert mixed symbols
text = "%10 indirimli ürün $50'ye satılıyor."
normalized = convert_symbols(text)
# Result: "yüzde 10 indirimli ürün 50 dolar'ye satılıyor."
```

--------------------------------

### Python: Create Custom SymbolConverter in trnorm

Source: https://github.com/ysdede/trnorm/blob/master/docs/README_symbols.md

Illustrates how to create a custom `SymbolConverter` instance from the `trnorm` package for advanced symbol conversion needs. This method allows disabling default mappings and adding custom ones programmatically.

```python
from trnorm import SymbolConverter

# Create a custom converter without loading default mappings
converter = SymbolConverter(load_defaults=False)

# Add custom mappings
converter.add_symbol_mapping("@", "et", False)
converter.add_symbol_mapping("#", "numara", False)

# Convert text using the custom converter
text = "@50 ve #20"
normalized = converter.convert_all_symbols(text)
# Result: "et 50 ve numara 20"
```

--------------------------------

### Create and Register Custom Transformer

Source: https://github.com/ysdede/trnorm/blob/master/docs/transformer.md

Illustrates how to define and register a custom transformer within the trnorm framework. This involves creating a Python function for the transformation logic and then using `create_custom_transformer` and `register_transformer`.

```python
from trnorm import create_custom_transformer, register_transformer

def my_custom_transformer(text):
    # Custom transformation logic
    return transformed_text

custom_transformer = create_custom_transformer(
    name='my_transformer',
    func=my_custom_transformer,
    description='My custom text transformation'
)
register_transformer(custom_transformer)
```
Tokens:
Raw