### Install neologdn

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Standard installation using pip. If build errors occur, try installing wheel first and then installing without build isolation.

```bash
pip install neologdn
```

```bash
pip install wheel
pip install --no-build-isolation neologdn
```

--------------------------------

### Install neologdn

Source: https://github.com/ikegami-yukino/neologdn/blob/master/README.md

Install the neologdn package using pip. If setuptools is not installed, install it first. If you encounter build isolation errors, try installing wheel and then installing neologdn with --no-build-isolation.

```sh
pip install neologdn
```

```sh
pip install setuptools
```

```sh
pip install wheel
pip install --no-build-isolation neologdn
```

--------------------------------

### Get CPU Brand String

Source: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

Retrieves the CPU brand string using sysctl. Useful for identifying the hardware.

```bash
sysctl -n machdep.cpu.brand_string
```

--------------------------------

### Benchmark neologdn Performance

Source: https://github.com/ikegami-yukino/neologdn/blob/master/README.md

Benchmark the performance of neologdn.normalize against a sample normalization function. This code requires the 'normalize_neologd' library to be installed.

```python
# Sample code from
# https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja#python-written-by-hideaki-t--overlast
import normalize_neologd

%timeit normalize(normalize_neologd.normalize_neologd)
# => 9.55 s ± 29.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

```python
import neologdn
%timeit normalize(neologdn.normalize)
# => 6.66 s ± 35.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

--------------------------------

### Benchmark Neologdn Normalization

Source: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

Measures the performance of the `neologdn.normalize` function using IPython's %timeit magic command. This requires the `neologdn` library to be installed.

```python
import neologdn
%timeit normalize(neologdn.normalize)
```

--------------------------------

### Benchmark Neologd Regex Normalization

Source: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

Measures the performance of the `normalize_neologd` function from the neologd project using IPython's %timeit magic command. This requires the `normalize_neologd` library to be installed.

```python
# Sample code from
# https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja#python-written-by-hideaki-t--overlast
import normalize_neologd

%timeit normalize(normalize_neologd.normalize_neologd)
```

--------------------------------

### Download and Prepare Benchmark Data

Source: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

Downloads the LDCC dataset, extracts it, and concatenates all text files into a single file for benchmarking. Ensure sufficient disk space.

```bash
curl -O https://www.rondhuit.com/download/ldcc-20140209.tar.gz
tar -x -f ldcc-20140209.tar.gz
find text/ -name "*.txt" -print0 | xargs -0 -I % cat %
 >> /tmp/ld.txt
```

--------------------------------

### Print Python Version

Source: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

Prints the current Python version. Useful for verifying the environment.

```python
import sys
print(sys.version)
```

--------------------------------

### Clean Up Benchmark Files

Source: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

Removes the downloaded dataset archive, extracted directory, and the concatenated text file to free up disk space.

```bash
rm -rf ldcc-20140209.tar.gz text ld.txt
```

--------------------------------

### Tilde Normalization Options

Source: https://github.com/ikegami-yukino/neologdn/blob/master/README.md

Customize tilde (〜) normalization using the 'tilde' parameter. Options include 'normalize' (convert to ~), 'normalize_zenkaku' (convert to 〜), 'ignore' (do not convert), and 'remove' (remove tildes). The default behavior is 'remove'.

```python
neologdn.normalize("1995〜2001年", tilde="normalize")
# => '1995~2001年'
```

```python
neologdn.normalize("1995~2001年", tilde="normalize_zenkaku")
# => '1995〜2001年'
```

```python
neologdn.normalize("1995〜2001年", tilde="ignore")  # Don't convert tilde
# => '1995〜2001年'
```

```python
neologdn.normalize("1995〜2001年", tilde="remove")
# => '19952001年'
```

```python
neologdn.normalize("1995〜2001年")  # Default parameter
# => '19952001年'
```

--------------------------------

### Benchmark Custom Normalization Overhead

Source: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

Measures the performance overhead of the `normalize` function itself, without any actual normalization logic, using IPython's %timeit magic command.

```python
def compute_overhead(x):
    pass

%timeit normalize(compute_overhead)
```

--------------------------------

### NLP Preprocessing Pipeline: Batch Processing with Error Handling

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Provides a robust way to normalize a list of texts, falling back to the original text if an error occurs during normalization. The `repeat` parameter can be adjusted for specific needs.

```python
import neologdn

# --- Batch processing with error handling ---
def safe_normalize(texts: list[str], **kwargs) -> list[str]:
    results = []
    for text in texts:
        try:
            results.append(neologdn.normalize(text, **kwargs))
        except Exception as e:
            results.append(text)  # fallback to original on error
    return results


corpus = ["ＮＬＰ処理", "テキスト　マイニング", "ディープラーニングーーー"]
normalized = safe_normalize(corpus, repeat=1)
# => ['NLP処理', 'テキストマイニング', 'ディープラーニング']
```

--------------------------------

### Preserve Full-width Exceptions

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Ensures specific full-width characters, like Japanese quotation marks, are preserved.

```python
# --- Full-width exceptions preserved ---
neologdn.normalize("全角記号例外「・」")
# => '全角記号例外「・」'
```

--------------------------------

### Normalize Half-width Katakana with Dakuten/Handakuten

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Handles dakuten (゛) and handakuten (゜) combining marks with half-width katakana.

```python
# --- Half-width katakana with dakuten/handakuten combining ---
neologdn.normalize("ﾊﾟﾊﾟ")   # ﾊ + ﾟ => パ
# => 'パパ'
neologdn.normalize("う゛ほﾟ")  # う + ゛ => ゔ, ほ + ﾟ => ぽ
# => 'ゔぽ'
```

--------------------------------

### Normalize Full-width ASCII and Symbols to Half-width

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Converts full-width ASCII characters and symbols to their standard half-width forms.

```python
# --- Full-width ASCII and symbols to half-width ---
neologdn.normalize("全角記号！？＠＃")
# => '全角記号!?@#'
```

--------------------------------

### NLP Preprocessing Pipeline: Basic Normalization

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Normalizes Japanese text for NLP tasks, including character conversion, repeat collapsing, whitespace removal, and tilde handling. The `tilde` parameter can be set to 'remove' or 'normalize'.

```python
import neologdn

def preprocess_for_nlp(text: str, keep_tilde_as_range: bool = False) -> str:
    """
    Normalize Japanese text before feeding into a morphological analyzer.

    - Converts all character variants to canonical forms
    - Collapses repeated characters (e.g., social media expressions)
    - Removes redundant whitespace
    - Handles tildes as range markers or removes them
    """
    tilde_mode = "normalize" if keep_tilde_as_range else "remove"
    return neologdn.normalize(
        text,
        repeat=3,                    # allow up to 3 consecutive repeats
        remove_space=True,           # strip spaces between Japanese tokens
        max_repeat_substr_length=8,  # check substrings up to length 8
        tilde=tilde_mode
    )


# Social media / user-generated content
samples = [
    "超うれしいいいいいいいいい！！！！！！！",
    "ﾏｼﾞｶ━━━━(ﾟ∀ﾟ)━━━━!!",
    "　　Ｐｙｔｈｏｎ　プログラミング　入門　",
    "明治時代（1868〜1912年）の出来事",
    "ありがとうございますすすすすすすすすす",
]

for s in samples:
    print(f"Input:  {s!r}")
    print(f"Output: {preprocess_for_nlp(s)!r}")
    print()

# Output:
# Input:  '超うれしいいいいいいいいい！！！！！！！'
# Output: '超うれしいいい!!!!!!'
#
# Input:  'ﾏｼﾞｶ━━━━(ﾟ∀ﾟ)━━━━!!'
# Output: 'マジカー(゚∀゚)ー!!'
#
# Input:  '　　Ｐｙｔｈｏｎ　プログラミング　入門　'
# Output: 'Pythonプログラミング入門'
#
# Input:  '明治時代（1868〜1912年）の出来事'
# Output: '明治時代(1868~1912年)の出来事'   # with keep_tilde_as_range=True
#
# Input:  'ありがとうございますすすすすすすすすす'
# Output: 'ありがとうございますすす'
```

--------------------------------

### neologdn.normalize()

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Normalizes a Japanese string by applying a comprehensive set of character-level transformations. This function handles half-width katakana to full-width, full-width ASCII/digits to ASCII, hyphen variants, long vowel marks, whitespace collapsing, dakuten/handakuten combining, and tilde handling. Optional parameters allow for controlling repeat-substring shortening, space removal, and tilde behavior.

```APIDOC
## neologdn.normalize()

### Description
Normalizes a Japanese string by applying a comprehensive set of character-level transformations: half-width katakana → full-width, full-width ASCII/digits → ASCII, hyphen variants → `-`, long vowel mark variants → `ー`, whitespace collapsing, dakuten/handakuten combining, and tilde handling. Accepts optional parameters to control repeat-substring shortening, space removal, and tilde behavior.

### Method
```python
neologdn.normalize(text: str, repeat: int = 0, remove_space: bool = True, tilde: str = 'remove') -> str
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **text** (str) - The input Japanese string to normalize.
- **repeat** (int, optional) - Shortens contiguous repeated single characters. Defaults to 0 (no shortening).
- **remove_space** (bool, optional) - If True, collapses redundant whitespace, including spaces between Japanese characters. Defaults to True.
- **tilde** (str, optional) - Controls how tilde variants are handled. Options are 'remove' (default), 'normalize' (to '~'), 'normalize_zenkaku' (to '〜'), or 'ignore'.

### Request Example
```python
import neologdn

# Basic normalization
print(neologdn.normalize("ﾊﾝｶｸｶﾅ"))
# => 'ハンカクカナ'

# With repeat parameter
print(neologdn.normalize("かわいいいいいいいいい", repeat=6))
# => 'かわいいいいいい'

# With tilde parameter
print(neologdn.normalize("1995〜2001年", tilde="normalize"))
# => '1995~2001年'
```

### Response
#### Success Response (200)
- **normalized_text** (str) - The normalized Japanese string.
```

--------------------------------

### Shorten Repeat: Long Sentence Repetition

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Demonstrates shortening repeated long sentences. The `repeat_threshold` parameter is crucial for controlling the reduction.

```python
from neologdn import shorten_repeat

# --- Long sentence repetition ---
sentence = '隣の客はよく柿食う客だ、隣の客はよく柿食う客だ、隣の客はよく柿食う客だ、言えた！'
shorten_repeat(sentence, 1, 0)
# => '隣の客はよく柿食う客だ、言えた！'
```

--------------------------------

### Normalize Tilde Variants

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Controls how different tilde variants (e.g., 〜, ~) are handled using the `tilde` parameter: 'remove' (default), 'normalize', 'normalize_zenkaku', or 'ignore'.

```python
# --- tilde='remove' (default): all tilde variants deleted ---
neologdn.normalize("1995〜2001年")
# => '19952001年'
```

```python
# --- tilde='normalize': all variants → ASCII tilde ~ ---
neologdn.normalize("1995〜2001年", tilde="normalize")
# => '1995~2001年'
```

```python
# --- tilde='normalize_zenkaku': all variants → full-width 〜 ---
neologdn.normalize("1995~2001年", tilde="normalize_zenkaku")
# => '1995〜2001年'
```

```python
# --- tilde='ignore': tildes left untouched ---
neologdn.normalize("1995〜2001年", tilde="ignore")
# => '1995〜2001年'
```

```python
# --- Tilde at boundary after Latin + space ---
neologdn.normalize("A ～あ", tilde="normalize")
# => 'A ~あ'
neologdn.normalize("A ～あ")  # tilde removed, space between Latin/JP also removed
```

--------------------------------

### Basic Japanese Text Normalization

Source: https://github.com/ikegami-yukino/neologdn/blob/master/README.md

Use the normalize function to perform basic Japanese text normalization, including converting half-width kana to full-width, normalizing symbols, shortening repeated characters, and removing tildes.

```python
import neologdn
neologdn.normalize("ﾊﾝｶｸｶﾅ")
# => 'ハンカクカナ'
```

```python
neologdn.normalize("全角記号！？＠＃")
# => '全角記号!?@#'
```

```python
neologdn.normalize("全角記号例外「・」")
# => '全角記号例外「・」'
```

```python
neologdn.normalize("長音短縮ウェーーーーイ")
# => '長音短縮ウェーイ'
```

```python
neologdn.normalize("チルダ削除ウェ~∼∾〜〰～イ")
# => 'チルダ削除ウェイ'
```

```python
neologdn.normalize("いろんなハイフン˗֊‐‑‒–⁃⁻₋−")
# => 'いろんなハイフン-'
```

```python
neologdn.normalize("　　　ＰＲＭＬ　　副　読　本　　　")
# => 'PRML副読本'
```

```python
neologdn.normalize(" Natural Language Processing ")
# => 'Natural Language Processing'
```

--------------------------------

### Normalize Half-width Katakana to Full-width

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Converts half-width katakana characters to their full-width equivalents.

```python
import neologdn

# --- Half-width katakana to full-width ---
neologdn.normalize("ﾊﾝｶｸｶﾅ")
# => 'ハンカクカナ'
```

--------------------------------

### neologdn.normalize

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Normalizes Japanese text by converting character variants to canonical forms, collapsing repeated characters, removing redundant whitespace, and handling tildes.

```APIDOC
## normalize()

### Description
Normalizes Japanese text for NLP preprocessing. This function handles character variants, collapses repeated characters, removes redundant whitespace, and manages tildes.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **text** (str) - Required - Input Japanese string
- **repeat** (int) - Optional - If > 0, shorten repeated substrings to at most `repeat` occurrences. `0` disables.
- **remove_space** (bool) - Optional - Remove spaces between Japanese (CJK/kana) characters. Defaults to True.
- **max_repeat_substr_length** (int) - Optional - Maximum substring length considered for repeat shortening. Defaults to 8.
- **tilde** (str) - Optional - Controls how tildes are handled. Options: `'remove'`, `'normalize'`, `'normalize_zenkaku'`, `'ignore'`. Defaults to `'remove'`.
```

--------------------------------

### Define Normalization Function

Source: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

Defines a placeholder function `normalize` that reads lines from a file. This function is used to measure the overhead of the normalization process itself.

```python
def normalize(func):
    with open('/tmp/ld.txt') as fd:
        for line in fd:
            func(line)
```

--------------------------------

### Handle Whitespace Between Japanese Characters

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Removes spaces between Japanese characters by default. Use `remove_space=False` to preserve them.

```python
# --- Whitespace handling: spaces between Japanese chars removed ---
neologdn.normalize("ゼンカク　スペース")
# => 'ゼンカクスペース'
neologdn.normalize("検索 エンジン 自作 入門 を 買い ました!!!")
# => '検索エンジン自作入門を買いました!!!'
```

```python
# --- Spaces between Latin words preserved ---
neologdn.normalize("Natural Language Processing")
# => 'Natural Language Processing'
```

```python
# --- Space between Latin and Japanese removed ---
neologdn.normalize("アルゴリズム C")
# => 'アルゴリズムC'
```

```python
# --- Leading/trailing whitespace stripped ---
neologdn.normalize(" Natural Language Processing ")
# => 'Natural Language Processing'
```

```python
# --- remove_space=False: preserve spaces between Japanese characters ---
neologdn.normalize("巴 マミ", remove_space=False)
# => '巴 マミ'
```

--------------------------------

### Normalize Full-width Digits and Letters

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Converts full-width digits and Latin letters to their standard forms, while stripping redundant whitespace.

```python
# --- Full-width digits and letters ---
neologdn.normalize("　　　ＰＲＭＬ　　副　読　本　　　")
# => 'PRML副読本'
```

--------------------------------

### Shorten Repeat: max_repeat_substr_length

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Illustrates how `max_repeat_substr_length` limits the inspection of substrings for repetition. A value of 0 means unlimited.

```python
from neologdn import shorten_repeat

# --- max_repeat_substr_length caps which substrings are inspected ---
# With max length = 11, the 12-char repeated sentence is NOT shortened
sentence = '隣の客はよく柿食う客だ、隣の客はよく柿食う客だ、隣の客はよく柿食う客だ、言えた！'
shorten_repeat(sentence, 1, 11)
# => '隣の客はよく柿食う客だ、隣の客はよく柿食う客だ、隣の客はよく柿食う客だ、言えた！'
```

--------------------------------

### Unify Hyphen Variants

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Normalizes various hyphen-like Unicode characters to the standard ASCII hyphen-minus.

```python
# --- Hyphen variants unified to ASCII hyphen-minus ---
neologdn.normalize("いろんなハイフン˗֊‐‑‒–⁃⁻₋−")
# => 'いろんなハイフン-'
```

--------------------------------

### Collapse Consecutive Long Vowel Marks

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Reduces multiple consecutive long vowel marks (ー) to a single instance.

```python
# --- Consecutive duplicate long vowel marks collapsed ---
neologdn.normalize("スーパーーーー")
# => 'スーパー'
```

--------------------------------

### Unify Long Vowel Mark Variants

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Converts different Unicode representations of the long vowel mark (chōonpu) to the standard ー character.

```python
# --- Long vowel mark (chōonpu) variants unified to ー ---
neologdn.normalize("majika━")
# => 'majikaー'
```

--------------------------------

### neologdn.shorten_repeat

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Shortens contiguous repeated substrings in a string independently of full normalization. Allows explicit control over the maximum substring length to inspect.

```APIDOC
## shorten_repeat()

### Description
Shortens contiguous repeated substrings in a string. This function can be called independently and supports explicit control over the maximum substring length to inspect.

### Parameters
- **text** (str) - Required - Input string
- **repeat_threshold** (int) - Required - Maximum number of allowed consecutive repetitions
- **max_repeat_substr_length** (int) - Required - Maximum length of substring to check for repetition; `0` means unlimited
```

--------------------------------

### Shorten Contiguous Repeated Substrings

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Limits the number of consecutive identical characters or substrings using the `repeat` parameter. A value of `n` keeps up to `n` repetitions.

```python
# --- repeat parameter: shorten contiguous repeated single characters ---
neologdn.normalize("かわいいいいいいいいい", repeat=6)
# => 'かわいいいいいい'  (up to 6 repetitions of 'い' kept)

neologdn.normalize("スーパーーーー", repeat=1)
# => 'スーパー'
```

```python
# --- repeat with multi-character substrings ---
neologdn.normalize("無駄無駄無駄無駄ァ", repeat=1)
# => '無駄ァ'
```

--------------------------------

### Shorten Repeated Characters

Source: https://github.com/ikegami-yukino/neologdn/blob/master/README.md

Control the shortening of contiguous repeated characters using the 'repeat' parameter. Set 'repeat' to a specific number to limit the repetitions, or to 1 to shorten all sequences of more than one repetition.

```python
neologdn.normalize("かわいいいいいいいいい", repeat=6)
# => 'かわいいいいいい'
```

```python
neologdn.normalize("無駄無駄無駄無駄ァ", repeat=1)
# => '無駄ァ'
```

--------------------------------

### Shorten Repeat: Multi-character Substring Repetition

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Shortens contiguous repetitions of multi-character substrings. The `repeat_threshold` parameter limits the occurrences of the substring.

```python
from neologdn import shorten_repeat

# --- Multi-character substring repetition ---
shorten_repeat('オラオラオラオラーッ', 2, 0)
# => 'オラオラーッ'  (keep up to 2 occurrences of 'オラ')

shorten_repeat('無駄無駄無駄無駄ァ', 1, 0)
# => '無駄ァ'  (keep only 1 occurrence of '無駄')
```

--------------------------------

### Shorten Repeat: Single Character Repetition

Source: https://context7.com/ikegami-yukino/neologdn/llms.txt

Shortens contiguous repetitions of single characters. The `repeat_threshold` parameter controls the maximum number of allowed consecutive repetitions.

```python
from neologdn import shorten_repeat

shorten_repeat('うまああああああああああああい', 7, 0)
# => 'うまあああああああい'  (keep up to 7 'あ')

# No change: 'い' appears 7 times but threshold is 6, count doesn't exceed threshold
shorten_repeat('かわいいいいいるい', 6, 0)
# => 'かわいいいいいるい'
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.