### Full Release Script Example

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

A consolidated script demonstrating the complete release process, from checking CI to verifying the installation.

```bash
VERSION="X.Y.Z"

# Verify CI is green
gh run list --limit 3

# Update versions (do this manually in your editor)
# - pyproject.toml
# - cleantext/__init__.py
# - CHANGELOG.md

# Commit, tag, push
git add pyproject.toml cleantext/__init__.py CHANGELOG.md
git commit -m "Release v${VERSION}"
git tag "v${VERSION}"
git push origin main --tags

# Create GitHub release
gh release create "v${VERSION}" --generate-notes

# Build and publish
poetry build
poetry publish

# Verify
pip install --upgrade clean-text
python -c "import cleantext; print(cleantext.__version__)"
```

--------------------------------

### Install with Both Extras

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Installs the clean-text library with both the GPL 'unidecode' and scikit-learn optional dependencies.

```bash
# With both
pip install clean-text[gpl,sklearn]
```

--------------------------------

### Install Clean-Text Core Library

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/README.md

Install the core clean-text library using pip.

```bash
# Core library
pip install clean-text
```

--------------------------------

### Verify Installation and Version

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Install the latest version of the package and print its version number to confirm the release.

```bash
pip install --upgrade clean-text
python -c "import cleantext; print(cleantext.__version__)"
```

--------------------------------

### Install Core Library

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Installs the core clean-text library without any optional dependencies.

```bash
# Core library (no GPL dependencies)
pip install clean-text
```

--------------------------------

### Install with Scikit-learn Support

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Installs the clean-text library with optional support for scikit-learn, requiring pandas, scikit-learn, and scipy.

```bash
# With scikit-learn support
pip install clean-text[sklearn]
```

--------------------------------

### Install with GPL Unidecode

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Installs the clean-text library with the optional GPL-licensed 'unidecode' package for improved transliteration.

```bash
# With GPL unidecode for better transliteration
pip install clean-text[gpl]
```

--------------------------------

### Install clean-text with GPL dependencies

Source: https://github.com/jfilter/clean-text/blob/main/README.md

Use this command to install the clean-text package along with its GPL-licensed dependencies, such as unidecode.

```bash
pip install clean-text[gpl]
```

--------------------------------

### Install clean-text without GPL dependencies

Source: https://github.com/jfilter/clean-text/blob/main/README.md

Use this command to install the clean-text package without including GPL-licensed dependencies. This is useful if you wish to avoid GPL licensing.

```bash
pip install clean-text
```

--------------------------------

### List Supported Languages and View Mappings

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/language-support.md

Demonstrates how to list all supported languages and inspect the character mappings for a specific language like German. Ensure the 'cleantext' library is installed.

```python
from cleantext.specials import specials_map

# List supported languages
print(specials_map.keys())
# dict_keys(['de', 'da', 'es', 'fo', 'fr', 'is', 'it', 'no', 'sv', 'se'])

# View German mappings
print(specials_map["de"])
# {'case_insensitive': [['ä', 'ae'], ...], 'case_sensitive': [...]}
```

--------------------------------

### Get CleanTransformer Parameters

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Retrieve the constructor parameters of the CleanTransformer as a dictionary. This is useful for inspecting the current configuration.

```python
cleaner = CleanTransformer(lower=False, no_punct=True)
cleaner.get_params()
# {'fix_unicode': True, 'to_ascii': True, ..., 'lower': False, 'no_punct': True, ...}
```

--------------------------------

### Example of Fenced Code Block

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/patterns-and-regex.md

Demonstrates a Python fenced code block with a simple function definition.

```python
```python
def hello():
    print("world")
```
```

--------------------------------

### Match Unix Relative File Paths

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/patterns-and-regex.md

Regex patterns for matching relative file paths starting with '~/', './', or '../'.

```regex
"~/...", "./...", "../..."
```

--------------------------------

### Python Exception Pattern Examples

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/patterns-and-regex.md

Illustrates various ways to define exception patterns using Python's 're' module syntax, including literal strings, character classes, hyphenated words, email addresses, dollar amounts, and multiple patterns.

```python
# Literal string
exceptions=["hello-world"]

# Character class
exceptions=[r"\w+"]  # Word characters

# Hyphenated words
exceptions=[r"\w+-\w+"]

# Email pattern
exceptions=[r"\S+@\S+"]

# Dollar amounts
exceptions=[r"\$\d+(\.\d{2})?"]

# Multiple patterns
exceptions=[r"\w+-\w+", r"\S+@\S+", r"\$\d+"]
```

--------------------------------

### ImportWarning for Missing unidecode

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/errors.md

This code demonstrates the import process when the optional 'unidecode' package is not installed. A warning is logged, and the library falls back to Python's 'unicodedata' for ASCII transliteration, which may yield lower quality results.

```python
# unidecode not installed
from cleantext import clean

# Warning is logged to stderr but processing continues
```

--------------------------------

### Match Windows Absolute File Paths

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/patterns-and-regex.md

Regex for matching absolute file paths in Windows systems, typically starting with a drive letter.

```regex
"C:\path\to\file"
```

--------------------------------

### Transliteration Fallback Warning

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

A warning message logged when the GPL-licensed 'unidecode' package is not installed and the library falls back to Python's 'unicodedata' for transliteration.

```text
WARNING:root:Since the GPL-licensed package `unidecode` is not installed,
using Python's `unicodedata` package which yields worse results.
```

--------------------------------

### Multiple Exception Patterns for Comprehensive Cleaning

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/common-usage-patterns.md

Combine multiple regex patterns in the `exceptions` list to preserve various elements like emails, monetary amounts, phone numbers, file paths, and hyphenated words simultaneously. This allows for fine-grained control over what gets cleaned and what is preserved.

```python
from cleantext import clean

text = """
Email: john-doe@example.com
Price: $49.99
Call: 555-1234
File: ./config/app.json
"""
result = clean(
    text,
    no_emails=True,
    no_currency_symbols=True,
    no_phone_numbers=True,
    no_file_paths=True,
    no_punct=True,
    exceptions=[
        r"\w+-\w+",           # hyphenated words
        r"\$\d+\.\d{2}",      # dollar amounts
        r"\d{3}-\d{4}",       # phone-like format (override removal)
        r"\./[\w/.]+"         # file paths (override removal)
    ]
)
# 'Email: john-doe@example.com Price: $49.99 Call: 555-1234 File: ./config/app.json'
```

--------------------------------

### Remove Social Media Noise with Exceptions

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/README.md

Example of removing URLs and emoji while preserving mentions and hashtags using custom regular expression exceptions.

```python
from cleantext import clean

text = "Check out @user's post: https://twitter.com/... #awesome 👍"
result = clean(
    text,
    no_urls=True,
    no_emoji=True,
    exceptions=[r"@\w+", r"#\w+"]  # Preserve mentions and hashtags
)
# 'check out @user's post: <url> #awesome'
```

--------------------------------

### Create GitHub Release with Notes

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Create a new GitHub release using the GitHub CLI, providing release notes via a heredoc.

```bash
gh release create vX.Y.Z --title "vX.Y.Z" --notes-file - <<'EOF'
## What's Changed

### Added
- Feature 1
- Feature 2

### Fixed
- Bug fix 1

**Full Changelog**: https://github.com/jfilter/clean-text/compare/vPREVIOUS...vX.Y.Z
EOF
```

--------------------------------

### Build and Publish to PyPI

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Build the package and publish it to the Python Package Index using Poetry.

```bash
poetry build
poetry publish
```

--------------------------------

### Create GitHub Release Interactively

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Create a new GitHub release using the GitHub CLI and automatically generate release notes.

```bash
gh release create vX.Y.Z --generate-notes
```

--------------------------------

### Initialize CleanTransformer with Parameters

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Instantiate CleanTransformer with various cleaning options. This is useful for setting up a reusable cleaning pipeline.

```python
from cleantext.sklearn import CleanTransformer

cleaner = CleanTransformer(
    fix_unicode=True,
    to_ascii=True,
    lower=True,
    no_urls=True,
    no_emails=True,
    no_punct=True,
    lang="en"
)
```

--------------------------------

### Project File Organization

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/INDEX.md

Overview of the project's directory structure and file placement.

```tree
output/
├── README.md                             # Start here
├── INDEX.md                              # This file
│
├── API REFERENCE
│   ├── api-reference-clean.md           # Main clean() function
│   ├── api-reference-clean-texts.md     # Batch clean_texts()
│   ├── api-reference-cleantransformer.md # SK-learn transformer
│   └── api-reference-helper-functions.md # Helper functions
│
├── CONFIGURATION & TYPES
│   ├── configuration.md                 # Parameters & settings
│   └── types.md                         # Type reference
│
├── TECHNICAL DETAILS
│   ├── module-overview.md               # Package structure
│   ├── patterns-and-regex.md            # Regex patterns
│   └── language-support.md              # Languages & mappings
│
└── REFERENCE
    ├── errors.md                        # Error handling
    └── common-usage-patterns.md         # Usage examples
```

--------------------------------

### Get Feature Names Out for CleanTransformer

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Return the output feature name for scikit-learn pipeline consistency. CleanTransformer always returns a single feature name.

```python
cleaner = CleanTransformer()
cleaner.get_feature_names_out()
# ['Clean Text']
```

--------------------------------

### Handling None Input and Default Exceptions

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/types.md

Demonstrates how the clean function handles None input by returning an empty string and how None is used as the default for the exceptions parameter.

```python
from cleantext import clean

# None input handling
clean(None)  # Returns ""

# None exceptions (default)
clean("text")  # Same as exceptions=None
```

--------------------------------

### Scikit-Learn Pipeline Integration

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/common-usage-patterns.md

Use CleanTransformer within a Scikit-Learn Pipeline for seamless integration into ML workflows. This example shows cleaning URLs and emails before vectorization and classification.

```python
from cleantext.sklearn import CleanTransformer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ('cleaner', CleanTransformer(no_urls=True, no_emails=True)),
    ('vectorizer', TfidfVectorizer(max_features=1000)),
    ('classifier', LogisticRegression()),
])

X_train = ["Visit https://example.com", "Email: test@example.com"]
y_train = [1, 0]

pipeline.fit(X_train, y_train)
predictions = pipeline.predict(["Email: another@test.com"])
```

--------------------------------

### Replacing URLs and Emails with Custom Tokens

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-clean.md

Illustrates how to replace URLs and email addresses with custom placeholder tokens using `replace_with_url` and `replace_with_email` parameters.

```python
from cleantext import clean

text = "Visit https://example.com or email contact@example.com"
clean(text, no_urls=True, no_emails=True, replace_with_url="[LINK]", replace_with_email="[MAIL]")
# 'visit [link] or email [mail]'
```

--------------------------------

### Optional Type Hints

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/types.md

Illustrates the conceptual use of optional types for parameters like text, exceptions, and n_jobs, which can accept None.

```python
# Equivalent to Optional[str] or str | None
text: str | None

# Equivalent to Optional[list[str]] or list[str] | None
exceptions: list[str] | None

# Equivalent to Optional[int] or int | None
n_jobs: int | None
```

--------------------------------

### Integrate Text Cleaning into Scikit-Learn Pipeline

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/README.md

The `CleanTransformer` can be used within a scikit-learn `Pipeline` for seamless integration into machine learning workflows. This example shows cleaning URLs before vectorization.

```python
from cleantext.sklearn import CleanTransformer
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ('cleaner', CleanTransformer(no_urls=True)),
    ('vectorizer', TfidfVectorizer()),
])
```

--------------------------------

### Commit, Tag, and Push Changes

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Stage changes, commit with a release message, tag the release, and push to the main branch.

```bash
git add pyproject.toml cleantext/__init__.py CHANGELOG.md
git commit -m "Release vX.Y.Z"
git tag vX.Y.Z
git push origin main --tags
```

--------------------------------

### Handle TypeError for invalid n_jobs type in clean_texts

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/errors.md

Catch and handle TypeError when the `n_jobs` parameter is not an integer or None. This example uses a try-except block to manage invalid type errors.

```python
from cleantext import clean_texts

try:
    clean_texts(texts, n_jobs="invalid")
except TypeError as e:
    print(f"Invalid n_jobs type: {e}")
```

--------------------------------

### Configure PyPI Token for Poetry

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Set the PyPI API token for publishing with Poetry, either directly or via an environment variable.

```bash
# Set your API token
poetry config pypi-token.pypi pypi-XXXXXXXXXXXX

# Or use environment variable
export POETRY_PYPI_TOKEN_PYPI=pypi-XXXXXXXXXXXX
poetry publish
```

--------------------------------

### Handle Invalid Regex Patterns in Exceptions

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/errors.md

Invalid regex patterns provided in the `exceptions` list will raise an error from the `re` module. This snippet shows how to catch such errors and demonstrates valid pattern examples.

```python
from cleantext import clean
import re

# Invalid regex
try:
    clean("text", exceptions=["[invalid"])  # Unclosed character class
except re.error as e:
    print(f"Invalid regex pattern: {e}")

# Valid patterns
clean("text", exceptions=[r"\w+-\w+"])  # Valid: word-word
clean("text", exceptions=["literal"])   # Valid: literal string
```

--------------------------------

### Fix ValueError for invalid input type in CleanTransformer

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/errors.md

Demonstrates how to fix ValueError by ensuring the input to `CleanTransformer.transform()` is either a list or a pandas Series. Includes examples for direct list usage, pandas Series, and converting iterables.

```python
from cleantext.sklearn import CleanTransformer
import pandas as pd

cleaner = CleanTransformer()

# Option 1: Use list
result = cleaner.transform(["text1", "text2"])

# Option 2: Use pandas Series
result = cleaner.transform(pd.Series(["text1", "text2"]))

# Option 3: Convert from iterable
texts = ["text1", "text2"]
result = cleaner.transform(list(texts))
```

--------------------------------

### Run clean-text benchmark

Source: https://github.com/jfilter/clean-text/blob/main/benchmarks/README.md

Execute the benchmark script for `clean_texts()` to compare sequential and parallel processing.

```bash
python benchmarks/bench_clean_texts.py
```

--------------------------------

### Running Type Checking with Mypy

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/types.md

Provides the command-line instruction for using mypy, a static type checker, to enforce type hints in Python code.

```bash
mypy your_code.py --ignore-missing-imports
```

--------------------------------

### Default Cleaning Profile Configuration

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Presents the default configuration for the clean function, detailing parameters for unicode fixing, ASCII transliteration, case normalization, whitespace handling, and removal of various elements like code, URLs, and punctuation.

```python
clean(
    text,
    fix_unicode=True,          # Fix mojibake
    to_ascii=True,             # Transliterate
    lower=True,                # Lowercase
    normalize_whitespace=True, # Normalize spaces
    no_line_breaks=False,      # Keep line breaks (normalized)
    strip_lines=True,          # Trim lines
    keep_two_line_breaks=False,# Collapse paragraphs
    no_code=False,             # Keep code
    no_urls=False,             # Keep URLs
    no_emails=False,           # Keep emails
    no_phone_numbers=False,    # Keep phone numbers
    no_ip_addresses=False,     # Keep IP addresses
    no_file_paths=False,       # Keep file paths
    no_numbers=False,          # Keep numbers
    no_digits=False,           # Keep digits
    no_currency_symbols=False, # Keep currency
    no_punct=False,            # Keep punctuation
    no_emoji=False,            # Keep emoji
    lang="en"                  # English handling
)
```

--------------------------------

### Import Core Cleaning Functions from cleantext

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/module-overview.md

Import the main cleaning functions directly from the cleantext package. These are the recommended imports for most use cases.

```python
from cleantext import clean
from cleantext import clean_texts
from cleantext import fix_bad_unicode
from cleantext import fix_strange_quotes
from cleantext import to_ascii_unicode
from cleantext import normalize_whitespace
from cleantext import replace_urls
from cleantext import replace_emails
from cleantext import replace_phone_numbers
from cleantext import replace_ip_addresses
from cleantext import replace_numbers
from cleantext import replace_digits
from cleantext import replace_currency_symbols
from cleantext import replace_code
from cleantext import replace_file_paths
from cleantext import remove_punct
from cleantext import replace_punct
from cleantext import remove_emoji
```

--------------------------------

### Regex Compilation Flags

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/patterns-and-regex.md

Demonstrates the flags used for compiling regular expressions at module import time. These flags control aspects like case sensitivity and line boundary matching.

```python
# Flags used
IGNORECASE = re.IGNORECASE  # Case-insensitive matching
UNICODE = re.UNICODE        # Unicode character classes
MULTILINE = re.MULTILINE    # ^ and $ match line boundaries
```

--------------------------------

### Clean-Text Default Parameters

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/README.md

Demonstrates the extensive list of default parameters available for the clean function, allowing fine-grained control over text cleaning processes.

```python
clean(
    text,
    fix_unicode=True,              # Fix mojibake
    to_ascii=True,                 # Transliterate
    lower=True,                    # Lowercase
    normalize_whitespace=True,     # Normalize spaces
    no_line_breaks=False,          # Keep normalized line breaks
    strip_lines=True,              # Trim lines
    keep_two_line_breaks=False,    # Collapse paragraphs
    no_code=False,                 # Keep code
    no_urls=False,                 # Keep URLs
    no_emails=False,               # Keep emails
    no_phone_numbers=False,        # Keep phone numbers
    no_ip_addresses=False,         # Keep IPs
    no_file_paths=False,           # Keep file paths
    no_numbers=False,              # Keep numbers
    no_digits=False,               # Keep digits
    no_currency_symbols=False,     # Keep currency
    no_punct=False,                # Keep punctuation
    no_emoji=False,                # Keep emoji
    replace_with_code="<CODE>",
    replace_with_url="<URL>",
    replace_with_email="<EMAIL>",
    replace_with_phone_number="<PHONE>",
    replace_with_ip_address="<IP>",
    replace_with_file_path="<FILE_PATH>",
    replace_with_number="<NUMBER>",
    replace_with_digit="0",
    replace_with_currency_symbol="<CUR>",
    replace_with_punct="",
    lang="en",
    exceptions=None
)
```

--------------------------------

### Basic Text Cleaning

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-clean.md

Demonstrates the default cleaning behavior of the `clean` function on a simple string.

```python
from cleantext import clean

text = "Hello   WORLD!!!"
clean(text)
# 'hello world'
```

--------------------------------

### Language-Specific Transliteration with to_ascii

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/language-support.md

Demonstrates how language-specific transliteration is applied when `to_ascii` is True and a supported `lang` is provided. Shows default behavior when `to_ascii` is False or an unsupported `lang` is used.

```python
from cleantext import clean

# German handling applied
clean("Äpfel", lang="de", to_ascii=True)  # 'aepfel'

# No German handling (generic transliteration)
clean("Äpfel", lang="de", to_ascii=False)  # 'Äpfel'

# Generic transliteration (English default)
clean("Äpfel", lang="en", to_ascii=True)  # depends on unidecode
```

--------------------------------

### Import from cleantext.utils

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/module-overview.md

Contains utility functions for text manipulation.

```APIDOC
## From cleantext.utils

```python
from cleantext.utils import remove_substrings
```

### `remove_substrings`

Removes specified substrings from a given string.
```

--------------------------------

### Comprehensive Text Cleaning with clean()

Source: https://github.com/jfilter/clean-text/blob/main/README.md

Demonstrates the extensive range of cleaning options available in the `clean` function, including unicode fixes, ASCII transliteration, case conversion, and removal of various elements like URLs, emails, and punctuation. Use this for general-purpose text cleaning.

```python
from cleantext import clean

clean("some input",
    fix_unicode=True,               # fix various unicode errors
    to_ascii=True,                  # transliterate to closest ASCII representation
    lower=True,                     # lowercase text
    no_line_breaks=False,           # fully strip line breaks as opposed to only normalizing them
    no_code=False,                  # replace all code snippets with a special token
    no_urls=False,                  # replace all URLs with a special token
    no_emails=False,                # replace all email addresses with a special token
    no_phone_numbers=False,         # replace all phone numbers with a special token
    no_ip_addresses=False,          # replace all IP addresses with a special token
    no_file_paths=False,            # replace all file paths with a special token
    no_numbers=False,               # replace all numbers with a special token
    no_digits=False,                # replace all digits with a special token
    no_currency_symbols=False,      # replace all currency symbols with a special token
    no_punct=False,                 # remove punctuations
    no_emoji=False,                 # remove emojis
    replace_with_punct="",          # instead of removing punctuations you may replace them
    exceptions=None,                # list of regex patterns to preserve verbatim
    replace_with_code="<CODE>",
    replace_with_url="<URL>",
    replace_with_email="<EMAIL>",
    replace_with_phone_number="<PHONE>",
    replace_with_ip_address="<IP>",
    replace_with_file_path="<FILE_PATH>",
    replace_with_number="<NUMBER>",
    replace_with_digit="0",
    replace_with_currency_symbol="<CUR>",
    lang="en"                       # set to 'de' for German special handling
)
```

--------------------------------

### German Language Text Conversion to ASCII

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-clean.md

Illustrates converting German special characters (like umlauts) to their closest ASCII equivalents using the `lang` and `to_ascii` parameters.

```python
from cleantext import clean

text = "Äpfel und Öl"
clean(text, lang="de", to_ascii=True)
# 'aepfel und oel'
```

--------------------------------

### Clean Text with Regex Exception

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Shows how to use a regular expression pattern to preserve multiple strings matching the pattern during text cleaning.

```python
# Regex pattern (raw string)
clean("hello-world and foo-bar", no_punct=True, exceptions=[r"\w+-\w+"])
```

--------------------------------

### Re-tagging a Release

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Instructions for deleting an existing local and remote tag and then re-tagging after making necessary fixes.

```bash
# Delete local and remote tag
git tag -d vX.Y.Z
git push origin :refs/tags/vX.Y.Z

# Re-tag after fixing
git tag vX.Y.Z
git push origin --tags
```

--------------------------------

### Match Currency Symbols

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/patterns-and-regex.md

Regex to match a predefined set of 15 supported currency symbols.

```regex
CURRENCY_REGEX
```

--------------------------------

### Profiling Performance of Batch Cleaning

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/common-usage-patterns.md

Profile the performance of batch cleaning with different `n_jobs` settings. Compare sequential and parallel execution times to determine the speedup achieved.

```python
from cleantext import clean_texts
import time

texts = ["Text " + str(i) for i in range(1000)]

start = time.time()
clean_texts(texts, n_jobs=1)
sequential_time = time.time() - start

start = time.time()
clean_texts(texts, n_jobs=-1)
parallel_time = time.time() - start

print(f"Sequential: {sequential_time:.3f}s")
print(f"Parallel: {parallel_time:.3f}s")
print(f"Speedup: {sequential_time / parallel_time:.2f}x")
```

--------------------------------

### Handling Phone Numbers and IP Addresses

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-clean.md

Demonstrates the removal of phone numbers and IP addresses using `no_phone_numbers` and `no_ip_addresses` parameters, replacing them with default tokens.

```python
from cleantext import clean

text = "Call 555-1234 or visit 192.168.1.1"
clean(text, no_phone_numbers=True, no_ip_addresses=True)
# 'call <phone> or visit <ip>'
```

--------------------------------

### ML Pipeline Integration with CleanTransformer

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/README.md

Illustrates integrating the CleanTransformer into a scikit-learn Pipeline for text cleaning followed by TF-IDF vectorization.

```python
from cleantext.sklearn import CleanTransformer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer

pipeline = Pipeline([
    ('cleaner', CleanTransformer(no_urls=True, no_emails=True)),
    ('vectorizer', TfidfVectorizer(max_features=1000)),
])

cleaned_vectors = pipeline.fit_transform(raw_texts)
```

--------------------------------

### Direct Imports from cleantext

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/module-overview.md

These functions are directly available for import from the top-level `cleantext` package and are recommended for general use.

```APIDOC
## Direct from cleantext

These are the recommended import paths for public API:

### `clean`

Cleans a string by applying various text normalization and cleaning techniques.

### `clean_texts`

Cleans a list of strings by applying various text normalization and cleaning techniques.

### `fix_bad_unicode`

Fixes common unicode errors in a string.

### `fix_strange_quotes`

Replaces various types of quotation marks with standard ASCII quotes.

### `to_ascii_unicode`

Converts unicode characters to their closest ASCII representation.

### `normalize_whitespace`

Normalizes whitespace in a string, replacing multiple spaces/tabs/newlines with a single space.

### `replace_urls`

Replaces all URLs in a string with a placeholder.

### `replace_emails`

Replaces all email addresses in a string with a placeholder.

### `replace_phone_numbers`

Replaces all phone numbers in a string with a placeholder.

### `replace_ip_addresses`

Replaces all IP addresses in a string with a placeholder.

### `replace_numbers`

Replaces all numbers in a string with a placeholder.

### `replace_digits`

Replaces all digits in a string with a placeholder.

### `replace_currency_symbols`

Replaces all currency symbols in a string with a placeholder.

### `replace_code`

Replaces code snippets in a string with a placeholder.

### `replace_file_paths`

Replaces file paths in a string with a placeholder.

### `remove_punct`

Removes all punctuation from a string.

### `replace_punct`

Replaces all punctuation in a string with a placeholder.

### `remove_emoji`

Removes all emoji characters from a string.
```

--------------------------------

### Clean Text with Multiple Regex Exceptions

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Illustrates cleaning text while preserving different types of patterns, such as email addresses and phone numbers, using multiple regex exceptions.

```python
# Multiple patterns
clean("test@example.com or 555-1234", exceptions=[r"\S+@\S+", r"\d+-\d+"])
```

--------------------------------

### Apply Specific Cleaning Options in Parallel

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-clean-texts.md

Clean texts in parallel while applying specific cleaning options like removing URLs and emails. The results are replaced with placeholder strings.

```python
from cleantext import clean_texts

texts = ["Visit https://example.com", "Email: test@example.com"]
clean_texts(texts, n_jobs=-1, no_urls=True, no_emails=True)
# ['visit <url>', 'email: <email>']
```

--------------------------------

### set_params(**params)

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Sets the constructor parameters for the CleanTransformer, allowing dynamic configuration of the cleaning process.

```APIDOC
## set_params(**params)

### Description
Set constructor parameters. Inherited from BaseEstimator.

### Parameters
- **params** (**params) - Parameter name-value pairs.

### Returns
- `self`

### Example
```python
cleaner = CleanTransformer()
cleaner.set_params(lower=False, no_urls=True)
```
```

--------------------------------

### GridSearchCV for CleanTransformer Parameter Tuning

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Demonstrates using GridSearchCV to tune CleanTransformer parameters within a scikit-learn pipeline. This is useful for optimizing cleaning steps alongside vectorization and classification.

```python
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from cleantext.sklearn import CleanTransformer

pipeline = Pipeline([
    ('cleaner', CleanTransformer()),
    ('vectorizer', TfidfVectorizer()),
    ('classifier', LogisticRegression()),
])

param_grid = {
    'cleaner__no_punct': [True, False],
    'cleaner__no_urls': [True, False],
    'vectorizer__max_features': [100, 500],
}

grid_search = GridSearchCV(pipeline, param_grid, cv=5)
```

--------------------------------

### Update Version Numbers

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Update the version in pyproject.toml and cleantext/__init__.py according to Semantic Versioning.

```markdown
## [Unreleased]

## [X.Y.Z] - YYYY-MM-DD

### Added
- ...

### Changed
- ...

### Fixed
- ...
```

--------------------------------

### clean() Function Signature

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-clean.md

This is the signature of the `clean()` function, detailing all available parameters and their default values for text preprocessing.

```python
def clean(
    text,
    fix_unicode=True,
    to_ascii=True,
    lower=True,
    normalize_whitespace=True,
    no_line_breaks=False,
    strip_lines=True,
    keep_two_line_breaks=False,
    no_code=False,
    no_urls=False,
    no_emails=False,
    no_phone_numbers=False,
    no_ip_addresses=False,
    no_file_paths=False,
    no_numbers=False,
    no_digits=False,
    no_currency_symbols=False,
    no_punct=False,
    no_emoji=False,
    replace_with_code="<CODE>",
    replace_with_url="<URL>",
    replace_with_email="<EMAIL>",
    replace_with_phone_number="<PHONE>",
    replace_with_ip_address="<IP>",
    replace_with_file_path="<FILE_PATH>",
    replace_with_number="<NUMBER>",
    replace_with_digit="0",
    replace_with_currency_symbol="<CUR>",
    replace_with_punct="",
    lang="en",
    exceptions=None,
) -> str
```

--------------------------------

### Testing Exception Patterns in Cleaning

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/common-usage-patterns.md

Test how specific patterns are handled during cleaning. Use the `exceptions` parameter to preserve patterns that would otherwise be removed.

```python
from cleantext import clean
import re

pattern = r"\w+-\w+"
test_text = "drive-thru and fly-by-night"

# Verify pattern matches
matches = re.findall(pattern, test_text)
print(f"Matches: {matches}")  # Should show hyphenated words

# Test with clean
result = clean(test_text, no_punct=True, exceptions=[pattern])
print(f"Result: {result}")
```

--------------------------------

### Set CleanTransformer Parameters

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Set constructor parameters for the CleanTransformer. This allows dynamic modification of the cleaning behavior.

```python
cleaner = CleanTransformer()
cleaner.set_params(lower=False, no_urls=True)
```

--------------------------------

### replace_file_paths()

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-helper-functions.md

Replaces file system paths (Unix and Windows, absolute and relative) with a specified replacement token. The default token is '<FILE_PATH>'.

```APIDOC
## replace_file_paths()

### Description
Replace file system paths with a specified token.

### Method Signature
```python
def replace_file_paths(text, replace_with="<FILE_PATH>") -> str
```

### Parameters
- **text** (str) - The input string to process.
- **replace_with** (str, optional) - The token to replace file paths with. Defaults to "<FILE_PATH>".

### Patterns Matched
- Unix absolute: /usr/local/bin, /home/user/file.txt
- Unix relative: ./src, ../lib, ~/Documents
- Windows absolute: C:\Windows\System32
- Windows relative: .\src, ..\lib

### Example
```python
from cleantext import replace_file_paths

text = "Check /usr/local/bin or C:\\Program Files"
replace_file_paths(text)
# 'Check <FILE_PATH> or <FILE_PATH>'
```
```

--------------------------------

### Batch Optimization with n_jobs

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/common-usage-patterns.md

Optimize batch text cleaning by choosing the `n_jobs` parameter wisely. For I/O intensive tasks, use more workers than CPU cores. For CPU intensive tasks, use the core count. For small batches, use sequential processing.

```python
from cleantext import clean_texts
import os

texts = ["Text " + str(i) for i in range(10000)]

# CPU cores available
cores = os.cpu_count()  # e.g., 8

# For I/O intensive: use more workers
results = clean_texts(texts, n_jobs=cores * 2)

# For CPU intensive: use core count
results = clean_texts(texts, n_jobs=cores)

# For small batches: use sequential
if len(texts) < 100:
    results = clean_texts(texts, n_jobs=1)
```

--------------------------------

### Import Utility Functions

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/module-overview.md

Import utility functions like `remove_substrings` from the cleantext.utils module. These functions provide specific text manipulation capabilities.

```python
from cleantext.utils import remove_substrings
```

--------------------------------

### Simple Text Normalization

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/common-usage-patterns.md

Use this for the most common use case of cleaning text with all default settings. It normalizes whitespace and converts text to lowercase.

```python
from cleantext import clean

text = "Hello   WORLD!!!  How are you?"
result = clean(text)
# 'hello world how are you'
```

--------------------------------

### Preserving Patterns with Exceptions

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-clean.md

Shows how to use the `exceptions` parameter to prevent specific patterns from being cleaned. This is useful for preserving hyphenated words or currency amounts.

```python
from cleantext import clean

# Preserve hyphenated compound words
clean("drive-thru and text---cleaning", no_punct=True, exceptions=[r"\w+-\w+"])
# 'drive-thru and textcleaning'

# Preserve literal currency amount
clean("Price is $99", no_currency_symbols=True, exceptions=[r"\$\d+"])
# 'price is $99'
```

--------------------------------

### Import Scikit-learn Transformer

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/module-overview.md

Import the CleanTransformer class for integration with scikit-learn pipelines. This allows using clean-text functionalities within machine learning workflows.

```python
from cleantext.sklearn import CleanTransformer
```

--------------------------------

### get_params(deep=True)

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Retrieves the constructor parameters of the CleanTransformer as a dictionary, useful for introspection and setting parameters.

```APIDOC
## get_params(deep=True)

### Description
Get constructor parameters as a dictionary. Inherited from BaseEstimator.

### Parameters
- **deep** (bool) - If True, recursively gets parameters of nested estimators.

### Returns
- Dictionary of parameter names and values.

### Example
```python
cleaner = CleanTransformer(lower=False, no_punct=True)
cleaner.get_params()
# {'fix_unicode': True, 'to_ascii': True, ..., 'lower': False, 'no_punct': True, ...}
```
```

--------------------------------

### GridSearchCV for Parameter Tuning

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/common-usage-patterns.md

Tune CleanTransformer parameters using GridSearchCV within a Scikit-Learn Pipeline. This allows for automated optimization of cleaning options alongside other pipeline steps.

```python
from cleantext.sklearn import CleanTransformer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ('cleaner', CleanTransformer()),
    ('vectorizer', TfidfVectorizer()),
    ('classifier', LogisticRegression()),
])

param_grid = {
    'cleaner__no_urls': [True, False],
    'cleaner__no_punct': [True, False],
    'vectorizer__max_features': [100, 500, 1000],
}

grid = GridSearchCV(pipeline, param_grid, cv=5)
grid.fit(X_train, y_train)
print(f"Best params: {grid.best_params_}")
```

--------------------------------

### Import from cleantext.sklearn

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/module-overview.md

Provides scikit-learn compatible transformers for text cleaning.

```APIDOC
## From cleantext.sklearn

```python
from cleantext.sklearn import CleanTransformer
```

### `CleanTransformer`

A scikit-learn transformer class for applying text cleaning operations within a pipeline.
```

--------------------------------

### Check CI Status

Source: https://github.com/jfilter/clean-text/blob/main/RELEASING.md

Verify that all tests are passing on the main branch before proceeding with the release.

```bash
gh run list --limit 5
```

--------------------------------

### Import Internal Constants and Specials

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/module-overview.md

Import internal constants and special mapping dictionaries. These are not recommended for direct use in production code as they may change without notice.

```python
from cleantext.constants import CURRENCIES, URL_REGEX, EMAIL_REGEX, ...
from cleantext.specials import specials_map, save_replace
```

--------------------------------

### Handle None Input Gracefully

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/errors.md

The `clean` function handles None input by returning an empty string. For explicit validation, a wrapper function can be used.

```python
from cleantext import clean

# None is handled gracefully
result = clean(None)  # Returns ""
assert result == ""

# But you might want to validate input first
def clean_safe(text):
    if text is None:
        return ""
    return clean(text)
```

--------------------------------

### Fit CleanTransformer

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Use the fit method for scikit-learn pipeline compatibility. This method does nothing as the transformer does not learn from data.

```python
from cleantext.sklearn import CleanTransformer
cleaner = CleanTransformer()
cleaner.fit(['text1', 'text2'])  # does nothing
```

--------------------------------

### Clean Text with Literal Exception

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/configuration.md

Demonstrates cleaning text while preserving a specific literal string that would otherwise be removed by punctuation rules.

```python
from cleantext import clean

# Literal string
clean("hello-world", no_punct=True, exceptions=["hello-world"])
```

--------------------------------

### Replace File Paths

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-helper-functions.md

Replace file system paths, including Unix and Windows formats (absolute and relative), with a specified token. The default token is '<FILE_PATH>'.

```python
from cleantext import replace_file_paths

text = "Check /usr/local/bin or C:\\Program Files"
replace_file_paths(text)
# 'Check <FILE_PATH> or <FILE_PATH>'
```

--------------------------------

### Basic Pipeline Integration with CleanTransformer

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Integrates CleanTransformer into a scikit-learn pipeline to remove URLs and emails before TF-IDF vectorization. Use this for standard text preprocessing workflows.

```python
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from cleantext.sklearn import CleanTransformer

pipeline = Pipeline([
    ('cleaner', CleanTransformer(no_urls=True, no_emails=True)),
    ('vectorizer', TfidfVectorizer()),
])

texts = ['Visit https://example.com', 'Email: test@example.com']
pipeline.fit_transform(texts)
```

--------------------------------

### Package Organization

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/module-overview.md

The clean-text package follows a standard Python module structure. Key components include the package entry point, core cleaning functions, Scikit-learn integration, constants, language-specific handling, and utility functions.

```text
cleantext/
├── __init__.py          # Package entry point (re-exports from clean)
├── clean.py             # Core cleaning functions
├── sklearn.py           # Scikit-learn transformer
├── constants.py         # Regex patterns and data
├── specials.py          # Language-specific handling
└── utils.py             # Utility functions
```

--------------------------------

### fit(X, y=None)

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Defined for scikit-learn pipeline compatibility. This method does nothing and is primarily for interface adherence.

```APIDOC
## fit(X, y=None)

### Description
Defined for scikit-learn pipeline compatibility. Does nothing.

### Parameters
- **X** (Any) - Input data (ignored)
- **y** (Any, optional) - Target data (ignored)

### Returns
- `self`

### Example
```python
from cleantext.sklearn import CleanTransformer
cleaner = CleanTransformer()
cleaner.fit(['text1', 'text2'])  # does nothing
```
```

--------------------------------

### Custom Cleaning Configuration in Pipeline

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Configures CleanTransformer with specific cleaning options and integrates it into a pipeline. Use this when you need fine-grained control over text cleaning for a specific language or set of rules.

```python
from cleantext.sklearn import CleanTransformer
from sklearn.pipeline import Pipeline

cleaner = CleanTransformer(
    fix_unicode=True,
    to_ascii=True,
    lower=True,
    no_urls=True,
    no_emails=True,
    no_phone_numbers=True,
    no_punct=True,
    lang='de'
)

pipeline = Pipeline([('clean', cleaner),])

result = pipeline.transform(['Text here'])
```

--------------------------------

### Clean Multiple Texts in Parallel (All Cores)

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-clean-texts.md

Process a list of texts in parallel using all available CPU cores. This is suitable for large datasets or computationally intensive cleaning tasks.

```python
from cleantext import clean_texts

texts = ["Text 1", "Text 2", "Text 3", "Text 4"]
clean_texts(texts, n_jobs=-1)
# Processes in parallel using all available CPU cores
```

--------------------------------

### Custom Replacement Tokens for Entities

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/common-usage-patterns.md

Replace removed entities like URLs and phone numbers with custom placeholder tokens. Specify your desired tokens using `replace_with_url` and `replace_with_phone_number`.

```python
from cleantext import clean

text = "Visit https://example.com or call 555-1234"
result = clean(
    text,
    no_urls=True,
    no_phone_numbers=True,
    replace_with_url="[LINK]",
    replace_with_phone_number="[PHONE]"
)
# 'visit [link] or call [phone]'
```

--------------------------------

### Match Unix Absolute File Paths

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/patterns-and-regex.md

Regex for matching absolute file paths in Unix-like systems, requiring at least two path segments.

```regex
/[\w.@+-]+
```

--------------------------------

### transform(X)

Source: https://github.com/jfilter/clean-text/blob/main/_autodocs/api-reference-cleantransformer.md

Applies text cleaning to each item in the input array-like, returning the cleaned text in the same format as the input.

```APIDOC
## transform(X)

### Description
Apply text cleaning to each item in the input array-like.

### Method
`transform`

### Parameters
- **X** (list[str] or pd.Series) - Array-like of strings to clean. Must be a Python list or pandas Series.

### Returns
- Same type as input (list or Series) with cleaned text.

### Raises
- `ValueError`: If X is not a list or pd.Series.

### Example
```python
from cleantext.sklearn import CleanTransformer

cleaner = CleanTransformer(no_punct=False, lower=False)
cleaner.transform(['Happily clean your text!', 'Another Input'])
# ['Happily clean your text', 'Another Input']

# With pandas Series
import pandas as pd
series = pd.Series(['Text 1', 'Text 2'])
cleaner.transform(series)  # Returns pd.Series
```
```