araclean

https://mhdmartini.github.io/araclean/latest
Admin
Araclean is a pure-Python library for Arabic text normalization and cleaning, fixing mojibake and...
Tokens:21,103
Snippets:292
Trust Score:9.7
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install araclean with Pandas or Polars Extra

Source: https://mhdmartini.github.io/araclean/latest/guides/dataframes

Install the necessary extra for pandas or polars integration. Use 'all' to install for both.

```bash
pip install 'araclean[pandas]'   # or 'araclean[polars]', or 'araclean[all]'
```

--------------------------------

### Install araclean

Source: https://mhdmartini.github.io/araclean/latest

Install the araclean library using pip. Optional extras like 'cli', 'pandas', 'polars', 'emoji', or 'all' can be installed for extended functionality.

```bash
pip install araclean

```

--------------------------------

### Default Knob Behavior Examples

Source: https://mhdmartini.github.io/araclean/latest/guides/tuning-profiles

Illustrates the default behavior of various knobs within different profiles, showcasing how they modify text normalization. Examples cover digit mapping, stopword removal, emoji handling, and elongation reduction.

```python
>>> normalize("مَدْرَسَةٌ", profile="search")                       # default: ة → ه
'مدرسه'
>>> normalize("مَدْرَسَةٌ", profile="search", teh_marbuta="keep")   # keep ة, still fold the rest
'مدرسة'
>>> normalize("كِتَابٌ", profile="ml", tashkeel_classes={"tanween"})  # remove only tanween
'كِتَاب'
>>> normalize("سطر١\nسطر٢", profile="search", collapse_lines=False)  # SEARCH, but keep lines
'سطر1\nسطر2'
```

--------------------------------

### Install Araclean with CLI support

Source: https://mhdmartini.github.io/araclean/latest/guides/cli

Install the araclean package with the 'cli' extra to enable command-line functionality.

```bash
pip install 'araclean[cli]'

```

--------------------------------

### Install Emoji Extra

Source: https://mhdmartini.github.io/araclean/latest/faq

To use the `HandleEmoji(mode="demojize")` functionality, which rewrites emojis to text aliases, you need to install the optional 'emoji' library.

```bash
pip install 'araclean[emoji]'
```

--------------------------------

### Install Araclean

Source: https://mhdmartini.github.io/araclean/latest/getting-started

Install the core araclean package using pip. This command installs the base library without any optional extras.

```bash
pip install araclean
```

--------------------------------

### Integrating Custom Step into a Pipeline

Source: https://mhdmartini.github.io/araclean/latest/guides/custom-steps

Shows how to add a custom step to an existing pipeline. A 'light' profile pipeline is loaded, and the custom 'StripQuestionMarks' step is appended. The example demonstrates running the pipeline and auditing its safety.

```python
light = Pipeline.from_profile("light")
pipe = Pipeline([*light.steps, StripQuestionMarks()])
pipe("كيف الحال؟")
pipe.audit().lossy_steps  # the audit sees your step's declared safety class
```

--------------------------------

### EmojiSupportNotInstalledError Class

Source: https://mhdmartini.github.io/araclean/latest/reference

Indicates that the emoji extra is not installed, which is required for `HandleEmoji(mode="demojize")`. This error subclasses ImportError.

```python
class EmojiSupportNotInstalledError(ImportError):
    """Raised when `HandleEmoji(mode="demojize")` is built without the optional `emoji` extra."""
    pass
```

--------------------------------

### Basic Text Normalization

Source: https://mhdmartini.github.io/araclean/latest/getting-started

Use the normalize function for basic text cleaning. This example shows how presentation forms and tatweel are handled.

```python
>>> from araclean import normalize
>>> normalize("ﻣﺮﺣﺒﺎ")  # OCR/copy-paste presentation forms fold back to real letters
'مرحبا'
>>> normalize("العـــربية")  # tatweel (visual elongation) is dropped
'العربية'
```

--------------------------------

### Get Offset Map with apply_aligned

Source: https://mhdmartini.github.io/araclean/latest/guides/offset-preserving

Use the `apply_aligned` method on a pipeline to get both the normalized text and an `OffsetMap`.

```python
normalized, omap = pipe.apply_aligned(text)
```

--------------------------------

### Check araclean Version

Source: https://mhdmartini.github.io/araclean/latest/guides/reproducibility

Verify the installed version of the araclean library. This is a basic check to ensure you are using a specific version for reproducibility.

```python
>>> import araclean
>>> isinstance(araclean.__version__, str)
True
```

--------------------------------

### Get translate_table for FoldHamza

Source: https://mhdmartini.github.io/araclean/latest/reference

This property provides the precomputed str.translate table for the FoldHamza step. It handles folding hamza off its carriers.

```python
translate_table: dict[int, str | None]

```

--------------------------------

### Get translate_table for FoldAlef

Source: https://mhdmartini.github.io/araclean/latest/reference

This property provides the static str.translate table applied by the FoldAlef step. It is used for folding alef variants to a bare alef.

```python
translate_table: dict[int, str]

```

--------------------------------

### Creating Custom Normalization Configurations

Source: https://mhdmartini.github.io/araclean/latest/guides/tuning-profiles

Demonstrates how to programmatically create and use a custom normalization configuration using `NormalizeConfig`. This is useful when overrides are insufficient and a custom pipeline is needed.

```python
>>> from araclean import NormalizeConfig, normalize
>>> config = NormalizeConfig(profile="ml", map_digits=True)
>>> normalize("٢٠٢٤", config=config)
'2024'
```

--------------------------------

### Create a Pipeline from a Profile

Source: https://mhdmartini.github.io/araclean/latest/guides/composing-pipelines

Construct a pipeline from a predefined profile and use it to process strings. Building the pipeline once and reusing it is the fast path.

```python
>>> from araclean import Pipeline
>>> pipe = Pipeline.from_profile("light")
>>> pipe
Pipeline([NormalizeUnicode, StripBidi, FoldPresentationForms, RemoveTatweel, UnifyLookalikes, CollapseWhitespace, NormalizeUnicode])
>>> pipe("العـــربية")
'العربية'
```

--------------------------------

### Build a Pipeline from Explicit Steps

Source: https://mhdmartini.github.io/araclean/latest/guides/composing-pipelines

Construct a pipeline by explicitly defining the sequence of steps. Steps are configured at construction time.

```python
>>> from araclean import CollapseWhitespace, NormalizeUnicode, RemoveTashkeel, Trim
>>> pipe = Pipeline([NormalizeUnicode(), RemoveTashkeel(), CollapseWhitespace(), Trim()])
>>> pipe("  مَرْحَبًا  بِكُمْ  ")
'مرحبا بكم'
```

--------------------------------

### PolarsExtraNotInstalledError Class

Source: https://mhdmartini.github.io/araclean/latest/reference

Raised when the polars namespace is accessed without the optional `[polars]` extra installed. This error subclasses ImportError.

```python
class PolarsExtraNotInstalledError(ImportError):
    """Raised when the polars namespace is used without the optional `[polars]` extra installed."""
    pass
```

--------------------------------

### PandasExtraNotInstalledError Class

Source: https://mhdmartini.github.io/araclean/latest/reference

Indicates that the pandas accessor is being used without the optional `[pandas]` extra installed. This error subclasses ImportError.

```python
class PandasExtraNotInstalledError(ImportError):
    """Raised when the pandas accessor is used without the optional `[pandas]` extra installed."""
    pass
```

--------------------------------

### CLIExtraNotInstalledError Class

Source: https://mhdmartini.github.io/araclean/latest/reference

Raised when the CLI is used without the optional `[cli]` extra (Typer) installed. This error subclasses ImportError.

```python
class CLIExtraNotInstalledError(ImportError):
    """Raised when the CLI is built without the optional `[cli]` extra (Typer) installed."""
    pass
```

--------------------------------

### Serialize and Restore Pipeline

Source: https://mhdmartini.github.io/araclean/latest/guides/reproducibility

Demonstrates how to serialize a complete preprocessing pipeline to a JSON-friendly dictionary and then restore it. This payload can be shipped with datasets to ensure exact preprocessing reproduction.

```python
>>> import json
>>> from araclean import Pipeline
>>> pipe = Pipeline.from_profile("search").drop("MapDigits")
>>> payload = json.dumps(pipe.to_dict(), ensure_ascii=False)  # ship this with your dataset
>>> restored = Pipeline.from_dict(json.loads(payload))
>>> restored("اَلسّلامُ عليكم") == pipe("اَلسّلامُ عليكم")
True
```

--------------------------------

### Normalization with Profiles

Source: https://mhdmartini.github.io/araclean/latest/getting-started

Apply different normalization profiles to tailor the cleaning process. Profiles like 'search', 'ml', and 'social' offer specific adjustments.

```python
>>> normalize("عَلَى")                    # LIGHT: vocalization and spelling preserved
'عَلَى'
>>> normalize("عَلَى", profile="search")  # SEARCH: tashkeel removed, alef maqsura folded
'علي'
>>> normalize("جميييييل", profile="ml")   # ML: dediacritize + collapse emphatic elongation
'جميل'
>>> normalize("رااااائع 😍 https://t.co/xyz", profile="social")  # SOCIAL: clean noise, keep emoji
'راائع 😍 [رابط]'
```

--------------------------------

### Custom Step with apply_aligned Hook

Source: https://mhdmartini.github.io/araclean/latest/guides/offset-preserving

Demonstrates how to create a custom normalization step that supports offset mapping by implementing the `apply_aligned` hook.

```python
from araclean import OffsetMap, SafetyClass

class MyStep:
    safety = SafetyClass.ENCODING_REPAIR

    def __call__(self, s: str, /) -> str:
        return s.replace("x", "y")  # 1→1 replace

    def apply_aligned(self, s: str, /) -> tuple[str, OffsetMap]:
        # 1→1 replacement: length is unchanged, each char maps to itself
        return self(s), OffsetMap.identity(len(s))
```

--------------------------------

### Adapt a Pipeline by Selecting Steps

Source: https://mhdmartini.github.io/araclean/latest/guides/composing-pipelines

Create a new pipeline containing only the specified steps in the given order. `select` is additive and useful for filtering and reordering.

```python
>>> Pipeline.from_profile("light").select("StripBidi", "RemoveTatweel")
Pipeline([StripBidi, RemoveTatweel])
```

--------------------------------

### Using Named Profiles with apply_aligned

Source: https://mhdmartini.github.io/araclean/latest/guides/offset-preserving

Shows how `apply_aligned` works seamlessly with pre-defined pipeline profiles like `SEARCH`.

```python
from araclean import Pipeline, SEARCH

pipe = Pipeline.from_profile(SEARCH)
normalized, omap = pipe.apply_aligned("وَقَالَ الرَّئيسُ")
```

--------------------------------

### Get translate_table for Trim Whitespace

Source: https://mhdmartini.github.io/araclean/latest/reference

This property provides the precomputed str.translate deletion table for the Trim Whitespace step. It is used when position is set to 'all'.

```python
translate_table: dict[int, None]

```

--------------------------------

### Pipeline From Profile Class Method

Source: https://mhdmartini.github.io/araclean/latest/reference

Build a pipeline from a named profile string (e.g., 'light') or a Profile object.

```python
from_profile(profile: str | Profile) -> Pipeline


```

--------------------------------

### Applying Per-Knob Overrides to Profiles

Source: https://mhdmartini.github.io/araclean/latest/guides/tuning-profiles

Demonstrates how to adjust specific behaviors of a chosen normalization profile using overrides. Overrides are validated upfront, failing loudly if a knob is misspelled or invalid for the profile.

```python
>>> from araclean import normalize
>>> normalize("٢٠٢٤", profile="ml")                    # ML keeps digits by default
'٢٠٢٤'
>>> normalize("٢٠٢٤", profile="ml", map_digits=True)   # opt in to the ASCII digit fold
'2024'
>>> normalize("جميييييل", profile="ml", elongation_cap=2)  # keep a doubled letter as emphasis
'جمييل'
```

--------------------------------

### Declaring Step Dependencies

Source: https://mhdmartini.github.io/araclean/latest/guides/custom-steps

Illustrates how to declare dependencies for a custom step using 'requires_before'. This ensures that specific prerequisite steps are present in the pipeline before the custom step is constructed, preventing invalid configurations.

```python
@dataclass(frozen=True)
class MyFoldedMatcher:
    requires_before: ClassVar[tuple[str, ...]] = ("RemoveTashkeel", "FoldAlef")
    ...
```

--------------------------------

### Adapt a Pipeline by Dropping Steps

Source: https://mhdmartini.github.io/araclean/latest/guides/composing-pipelines

Create a new pipeline by removing specific steps from an existing profile. `drop` removes every step carrying a name and raises `KeyError` for a name no step carries.

```python
>>> Pipeline.from_profile("search").drop("MapDigits", "MapPunctuation")
Pipeline([NormalizeUnicode, StripBidi, FoldPresentationForms, RemoveTatweel, UnifyLookalikes, CollapseWhitespace, NormalizeUnicode, FoldTanweenAlef, RemoveTashkeel, FoldAlef, FoldHamza, FoldTehMarbuta, FoldAlefMaqsura, ReduceElongation, CollapseWhitespace, NormalizeUnicode])
```

--------------------------------

### RAG: Cite in Original, Search in Normalized

Source: https://mhdmartini.github.io/araclean/latest/guides/offset-preserving

Demonstrates using `apply_aligned` for RAG. Normalize text for indexing and use the `OffsetMap` to project search hits back to the original text for citation.

```python
from araclean import Pipeline, RemoveTatweel, FoldAlef, RemoveTashkeel

pipe = Pipeline([RemoveTatweel(), FoldAlef(), RemoveTashkeel()])

# Index-time: normalize, store the original
original = "كتاب أحمـد الكبير"
normalized, omap = pipe.apply_aligned(original)

# Retrieval-time: a search hit in the normalized index gives a span
# Suppose a fuzzy search found "احمد" somewhere in the normalized text
found_start = normalized.index("احمد")
found_end = found_start + len("احمد")

# Project back to the original for citation
orig_start, orig_end = omap.to_original((found_start, found_end))
citation = original[orig_start:orig_end]
print(citation)   # "أحمـد" — the original spelling, with tatweel and hamza intact
```

--------------------------------

### Pipeline Construction with Step Dependencies

Source: https://mhdmartini.github.io/araclean/latest/guides/composing-pipelines

Demonstrates a `ValueError` raised during pipeline construction if a step's required preceding steps are not met. This check happens at construction, not per string.

```python
>>> from araclean import RemoveStopwords
>>> Pipeline([RemoveStopwords()])
Traceback (most recent call last):
  ...
ValueError: Step 'RemoveStopwords' requires ['RemoveTashkeel', 'FoldAlef', 'FoldAlefMaqsura', 'FoldHamza'] to run before it in the pipeline ...
```

--------------------------------

### Serialize and Validate Normalization Configuration

Source: https://mhdmartini.github.io/araclean/latest/guides/reproducibility

Illustrates serializing a `NormalizeConfig` object to JSON and then validating a JSON string against this configuration model. This ensures tuned calls are reproducible and validate correctly.

```python
>>> from araclean import NormalizeConfig, normalize
>>> config = NormalizeConfig(profile="ml", map_digits=True)
>>> config.model_dump_json(exclude_defaults=True)
'{"profile":"ml","map_digits":true}'
>>> normalize("٢٠٢٤", config=NormalizeConfig.model_validate_json('{"profile":"ml","map_digits":true}'))
'2024'
```

--------------------------------

### Assemble Effective Profile

Source: https://mhdmartini.github.io/araclean/latest/reference

Assembles the final Profile by applying configuration overrides to a named preset. Raises ValueError for invalid overrides or unknown profiles.

```python
resolve() -> Profile
```

--------------------------------

### Pipeline Select Method

Source: https://mhdmartini.github.io/araclean/latest/reference

Create a new pipeline containing only the specified steps in the given order. Used for filtering or reordering steps.

```python
select(*names: str) -> Pipeline


```

--------------------------------

### Minimal Custom Step Implementation

Source: https://mhdmartini.github.io/araclean/latest/guides/custom-steps

Defines a basic custom step 'StripQuestionMarks' that removes question marks. It demonstrates the required 'safety' attribute and the '__call__' method signature. This step can be directly used in a Pipeline.

```python
from dataclasses import dataclass
from araclean import Pipeline, SafetyClass, Step

@dataclass(frozen=True)
class StripQuestionMarks:
    safety = SafetyClass.LINGUISTIC_FOLDING
    def __call__(self, s: str, /) -> str:
        return s.replace("؟", "").replace("?", "")

isinstance(StripQuestionMarks(), Step)  # a runtime-checkable Protocol — no base class needed
```

--------------------------------

### Custom Step for Fused Engine Optimization

Source: https://mhdmartini.github.io/araclean/latest/guides/custom-steps

Shows how a custom step can opt into the fused engine by implementing a 'translate_table' property. This allows steps performing static table translations to be combined into a single C-level pass for performance gains.

```python
@dataclass(frozen=True)
class MyFold:
    safety = SafetyClass.LINGUISTIC_FOLDING

    @property
    def translate_table(self) -> dict[int, str | None]:
        return self._table  # what __call__ does: s.translate(self._table)

    def __call__(self, s: str, /) -> str:
        return s.translate(self._table)
```

--------------------------------

### Pipeline To Dict Method

Source: https://mhdmartini.github.io/araclean/latest/reference

Serialize the pipeline into a JSON-friendly dictionary format. Raises an error if any step cannot serialize itself.

```python
to_dict() -> PipelineDict


```

--------------------------------

### One-off String Transformation with Bare Functions

Source: https://mhdmartini.github.io/araclean/latest/guides/composing-pipelines

Use plain functions for single transformations without building a pipeline. These are the hot path called by steps.

```python
>>> from araclean import fold_alef, map_digits, remove_tatweel
>>> remove_tatweel("محـــمد")
'محمد'
>>> fold_alef("أحمد إلى آخر")
'احمد الى اخر'
>>> map_digits("٠١٢٣٤٥٦٧٨٩")
'0123456789'
```

--------------------------------

### Pipeline Apply Aligned Method

Source: https://mhdmartini.github.io/araclean/latest/reference

Normalize text while tracking the mapping of positions back to the original text. Returns the normalized text and an offset map.

```python
apply_aligned(text: str) -> tuple[str, OffsetMap]


```

--------------------------------

### Normalize digits using the 'ml' profile

Source: https://mhdmartini.github.io/araclean/latest/guides/cli

Apply the 'ml' profile to normalize digits in the input text.

```bash
$ printf '٢٠٢٤\n' | araclean normalize -p ml --map-digits
2024

```

--------------------------------

### Preserve Teh Marbuta using the 'search' profile

Source: https://mhdmartini.github.io/araclean/latest/guides/cli

Use the 'search' profile and the '--teh-marbuta keep' option to preserve Teh Marbuta characters.

```bash
$ printf 'كِتَابٌ في مَدْرَسَةٍ\n' | araclean normalize -p search --teh-marbuta keep
كتاب في مدرسة

```

--------------------------------

### Embed Configuration in Pipeline Step

Source: https://mhdmartini.github.io/araclean/latest/guides/reproducibility

Shows how a specific configuration, like the version of a stopword list used by `RemoveStopwords`, is embedded within the serialized form of the pipeline step.

```python
>>> from araclean import RemoveStopwords
>>> RemoveStopwords().to_dict()
{'name': 'RemoveStopwords', 'config': {'version': '2.0.0'}}
```

--------------------------------

### Import and Inspect Stopword List

Source: https://mhdmartini.github.io/araclean/latest/guides/stopwords

Access the bundled stopword list, negation particles, and version information directly from the `araclean.stopwords` module. Verify that the stopword list is disjoint from negation particles.

```python
>>> from araclean.stopwords import NEGATION_PARTICLES, STOPWORDS, STOPWORDS_VERSION
>>> "في" in STOPWORDS
True
>>> STOPWORDS.isdisjoint(NEGATION_PARTICLES)
True
```

--------------------------------

### Audit Pipeline Losslessness and Lossy Steps

Source: https://mhdmartini.github.io/araclean/latest/guides/composing-pipelines

Report whether a pipeline is lossless and identify which steps cause data loss. This helps in understanding the impact of preprocessing.

```python
>>> report = Pipeline.from_profile("search").audit()
>>> report.lossless
False
>>> report.lossy_steps
('FoldTanweenAlef', 'RemoveTashkeel', 'FoldAlef', 'FoldHamza', 'FoldTehMarbuta', 'FoldAlefMaqsura', 'MapDigits', 'MapPunctuation', 'ReduceElongation')
```

--------------------------------

### Normalize Arabic Text (Search Profile)

Source: https://mhdmartini.github.io/araclean/latest

Utilize the 'SEARCH' profile for normalization that maximizes recall by folding spelling variants. This profile is lossy.

```python
normalize("اَلسّلامُ عليكم", profile="search")  # SEARCH: lossy folds that maximize recall

```

--------------------------------

### Keep Specific Marks with Knobs

Source: https://mhdmartini.github.io/araclean/latest/faq

When using a lossy profile, you can retain specific characters like 'teh marbuta' or control 'tashkeel' classes using profile knobs instead of dropping entire steps.

```python
teh_marbuta="keep", tashkeel_classes={...}
```

--------------------------------

### Drop UnifyLookalikes Step

Source: https://mhdmartini.github.io/araclean/latest/faq

If the automatic unification of look-alike characters (like Farsi yeh and alef maqsura) is problematic for your corpus, you can explicitly drop this step from the LIGHT profile.

```python
Pipeline.from_profile("light").drop("UnifyLookalikes")
```

--------------------------------

### Normalize Polars Series

Source: https://mhdmartini.github.io/araclean/latest/guides/dataframes

Import araclean.polars to register the namespace and normalize a polars Series. Null values pass through unchanged.

```python
>>> import polars as pl
>>> import araclean.polars
>>> s = pl.Series(["العـــربية", "اَلسّلامُ عليكم"])
>>> s.araclean.normalize(profile="search").to_list()
['العربيه', 'السلام عليكم']
```

--------------------------------

### SafetyReport lossy_steps property

Source: https://mhdmartini.github.io/araclean/latest/reference

Lists all lossy steps in a pipeline, including linguistic folds and cleaning steps. This helps identify where information might be lost.

```python
lossy_steps: tuple[str, ...]
```

--------------------------------

### Normalize text from a file to a file

Source: https://mhdmartini.github.io/araclean/latest/guides/cli

Process a text file by specifying input and output file paths for normalization.

```bash
$ araclean normalize corpus.txt --profile search --output corpus.clean.txt

```

--------------------------------

### Pipeline From Dict Class Method

Source: https://mhdmartini.github.io/araclean/latest/reference

Rehydrate a pipeline from a dictionary representation using the step registry. This is the inverse of `to_dict`.

```python
from_dict(data: Mapping[str, Any]) -> Pipeline


```

--------------------------------

### Process a Batch of Strings with a Pipeline

Source: https://mhdmartini.github.io/araclean/latest/guides/composing-pipelines

Use the `batch` method for lazy generation of processed strings from a list. Nothing is materialized until iterated.

```python
>>> list(pipe.batch(["العـــربية", "ﻣﺮﺣﺒﺎ"]))
['العربية', 'مرحبا']
```

--------------------------------

### Compose Offset Maps

Source: https://mhdmartini.github.io/araclean/latest/reference

Chains two OffsetMaps together. Use this to accumulate transformations by composing subsequent maps onto a running map.

```python
compose(other: OffsetMap) -> OffsetMap
```

--------------------------------

### Build Offset Map from Regex Substitution

Source: https://mhdmartini.github.io/araclean/latest/reference

Generates an OffsetMap based on match spans from a regex substitution. Each matched segment in the original string is mapped back to its corresponding replacement length in the normalized text.

```python
from_regex_sub(
    original: str,
    match_spans: Sequence[tuple[int, int]],
    replacement_lens: Sequence[int],
) -> OffsetMap
```

--------------------------------

### NER: Project Model Output to Original Text

Source: https://mhdmartini.github.io/araclean/latest/guides/offset-preserving

Illustrates using `apply_aligned` for NER. A model predicts spans on normalized text; the `OffsetMap` projects these spans back to the original document.

```python
from araclean import Pipeline, FoldAlef, RemoveTashkeel, RemoveTatweel

pipe = Pipeline([RemoveTatweel(), RemoveTashkeel(), FoldAlef()])

original_doc = "قال الرئيسُ محمـدٌ في المؤتمرِ"
normalized, omap = pipe.apply_aligned(original_doc)

# A NER model running on normalized text predicts a PERSON span
ner_start = normalized.index("محمد")
ner_end = ner_start + len("محمد")

# Project to original text
orig_start, orig_end = omap.to_original((ner_start, ner_end))
original_span = original_doc[orig_start:orig_end]
print(original_span)   # "محمـد" — the original spelling, tatweel intact (the deleted
                       # trailing tanween sits past the span: it produced no normalized char)
```

--------------------------------

### Normalize Arabic Text (Default Profile)

Source: https://mhdmartini.github.io/araclean/latest

Use the normalize function for basic Arabic text normalization. The default 'LIGHT' profile performs lossless encoding repair, which may drop characters like tatweel.

```python
from araclean import normalize
normalize("العـــربية")  # default LIGHT profile: lossless encoding repair (here, drops tatweel)

```

--------------------------------

### Validate Configuration Overrides

Source: https://mhdmartini.github.io/araclean/latest/concepts/safety

Demonstrates how Araclean raises a ValueError for invalid configuration overrides that do not apply to the selected profile, ensuring loud failure.

```python
>>> from araclean import normalize
>>> try:
...     normalize("نص", profile="light", emoji="strip")
... except ValueError as error:
...     print(error)
override(s) ['emoji'] do not apply to profile 'light': it has no matching step to configure.
```

--------------------------------

### OffsetMap

Source: https://mhdmartini.github.io/araclean/latest/reference

Represents the alignment from normalized-text positions back to original-text positions, returned by `Pipeline.apply_aligned`.

```APIDOC
## OffsetMap

### Description
Alignment from normalized-text positions back to original-text positions.

Internally stores a flat `list[int]` of length `2 * len(normalized)`: entries `[2*i, 2*i+1]` are `(orig_start, orig_end)` for normalized character `i`.
```

--------------------------------

### Build Identity Map

Source: https://mhdmartini.github.io/araclean/latest/reference

Creates an identity map where each normalized character maps to its original position. Use this when no translation or transformation is needed.

```python
identity(n: int) -> OffsetMap
```

--------------------------------

### Importing Stopwords Data

Source: https://mhdmartini.github.io/araclean/latest/reference

Import necessary stopword data and configuration constants from the araclean module.

```python
from araclean import STOPWORDS, STOPWORDS_VERSION, STOPWORDS_LICENSE, NEGATION_PARTICLES
```

--------------------------------

### Pipeline

Source: https://mhdmartini.github.io/araclean/latest/reference

Represents an ordered sequence of normalization steps. It can be called directly to process text or used to build custom pipelines.

```APIDOC
## Pipeline

### Description
An ordered, serializable sequence of `Step`s, callable like a single `str -> str`.

### Attributes
* **steps** (tuple[Step, ...]) - The ordered steps, for inspection.

### Methods
* **batch(texts: Iterable[str]) -> Iterator[str]**
  Normalize each text lazily — a streaming generator.
* **select(*names: str) -> Pipeline**
  Build a NEW pipeline holding exactly the named steps, in the order given.
* **drop(*names: str) -> Pipeline**
  Build a NEW pipeline WITHOUT every step matching each name.
* **audit() -> SafetyReport**
  Audit this pipeline's safety: is it lossless, and if not, what it loses.
* **apply_aligned(text: str) -> tuple[str, OffsetMap]**
  Normalize _text_ while tracking how every position maps back to the original.
* **to_dict() -> PipelineDict**
  Serialize to a plain, JSON-friendly dict.

### Class Methods
* **from_dict(data: Mapping[str, Any]) -> Pipeline**
  Rehydrate a pipeline from `to_dict()` output via the step registry.
* **from_profile(profile: str | Profile) -> Pipeline**
  Build a pipeline from a named profile (e.g. "light") or a `Profile` object.
```

--------------------------------

### Custom Step with Serialization Support

Source: https://mhdmartini.github.io/araclean/latest/guides/custom-steps

Implements a custom step 'MaskNumbers' that masks digits with '#'. It includes 'to_dict' and 'from_dict' methods for serialization and deserialization, and is registered with the Araclean registry. This allows pipelines containing this step to be saved and loaded.

```python
from typing import ClassVar
from araclean.registry import register

@dataclass(frozen=True)
class MaskNumbers:
    safety = SafetyClass.LINGUISTIC_FOLDING
    name: ClassVar[str] = "MaskNumbers"
    def __call__(self, s: str, /) -> str:
        return "".join("#" if ch.isdigit() else ch for ch in s)
    def to_dict(self):
        return {"name": self.name, "config": {}}
    @classmethod
    def from_dict(cls, config):
        return cls(**config)

register(MaskNumbers.name, MaskNumbers.from_dict)
pipe = Pipeline([MaskNumbers()])
rebuilt = Pipeline.from_dict(pipe.to_dict())
rebuilt("غرفة 101")
```

--------------------------------

### Apply Aligned Normalization with Custom Pipeline

Source: https://mhdmartini.github.io/araclean/latest

Create a custom pipeline with specific steps like RemoveTatweel and FoldAlef for span-level text processing. The apply_aligned method returns the normalized text and a map to original positions.

```python
from araclean import Pipeline, RemoveTatweel, FoldAlef
pipe = Pipeline([RemoveTatweel(), FoldAlef()])
normalized, omap = pipe.apply_aligned("أحمـد")
normalized

```

```python
omap.to_original((0, 4))   # where does the whole normalized word sit in the original?

```

--------------------------------

### normalize

Source: https://mhdmartini.github.io/araclean/latest/reference

The main facade function to normalize Arabic text. It accepts text and optional configuration profiles or overrides to customize the normalization process.

```APIDOC
## normalize

### Description
Normalize Arabic text with a named profile (default `LIGHT` — lossless encoding repair).

### Method Signature
```python
normalize(
    text: str,
    *, 
    profile: str | Profile | None = None,
    config: NormalizeConfig | None = None,
    **overrides: object,
) -> str
```

### Parameters
* **text** (str) - The input Arabic text to normalize.
* **profile** (str | Profile | None, optional) - The name of a predefined normalization profile (e.g., "search", "ml"), a `Profile` object, or `None` to use the default `LIGHT` profile. Defaults to `None`.
* **config** (NormalizeConfig | None, optional) - A `NormalizeConfig` object for fully custom configuration. Cannot be used with `profile` or `overrides`. Defaults to `None`.
* **overrides** (object) - Keyword arguments to override specific settings within a named profile (e.g., `map_digits=True`).

### Returns
- **str** - The normalized Arabic text.
```

--------------------------------

### Pipeline Steps Property

Source: https://mhdmartini.github.io/araclean/latest/reference

Access the ordered sequence of steps within a Pipeline object for inspection.

```python
steps: tuple[Step, ...]


```

--------------------------------

### Handle invalid profile overrides

Source: https://mhdmartini.github.io/araclean/latest/guides/cli

The CLI rejects overrides that do not apply to the selected profile, preventing silent errors.

```bash
$ printf 'نص\n' | araclean normalize -p light --emoji strip
error: override(s) ['emoji'] do not apply to profile 'light': it has no matching step to configure.

```

--------------------------------

### Generate JSON Schema for Normalization Configuration

Source: https://mhdmartini.github.io/araclean/latest/guides/reproducibility

Retrieves the JSON schema for the `NormalizeConfig` model. This schema can be used by non-Python tooling to validate configurations.

```python
>>> schema = NormalizeConfig.model_json_schema()
>>> schema["title"], schema["additionalProperties"]
('NormalizeConfig', False)
```

--------------------------------

### Pipeline Drop Method

Source: https://mhdmartini.github.io/araclean/latest/reference

Create a new pipeline excluding steps that match the given names. This is a subtractive adapter for modifying pipelines.

```python
drop(*names: str) -> Pipeline


```

--------------------------------

### Normalize text from stdin to stdout

Source: https://mhdmartini.github.io/araclean/latest/guides/cli

Use 'araclean normalize' to read from standard input and write normalized text to standard output, suitable for piping.

```bash
$ printf 'اَلسّلامُ عليكم\nالعـــربية\n' | araclean normalize --profile search
السلام عليكم
العربيه

```

--------------------------------

### Build Offset Map from Translation Table

Source: https://mhdmartini.github.io/araclean/latest/reference

Constructs an OffsetMap from an original string and its str.translate table. This method handles various translation operations including deletions, replacements, and expansions.

```python
from_translate(
    original: str, table: Mapping[int, str | int | None]
) -> OffsetMap
```

--------------------------------

### Normalize Function Signature

Source: https://mhdmartini.github.io/araclean/latest/reference

The main facade function for normalizing Arabic text. It accepts text and optional configuration parameters like profile, config, and overrides.

```python
normalize(
    text: str,
    *,
    profile: str | Profile | None = None,
    config: NormalizeConfig | None = None,
    **overrides: object,
) -> str


```

--------------------------------

### StripBidi dataclass

Source: https://mhdmartini.github.io/araclean/latest/reference

Removes bidi controls, zero-width characters, and the BOM. This is a lossless encoding repair step that cleans invisible characters that can interfere with text processing.

```python
class StripBidi:
    """Remove bidi controls, zero-width characters and the BOM — lossless encoding repair."""
    pass
```

--------------------------------

### Audit Pipeline for Data Loss

Source: https://mhdmartini.github.io/araclean/latest/guides/reproducibility

Generates an audit report for a pipeline to determine if any data was lost during preprocessing and identifies the specific steps that caused data loss.

```python
>>> report = Pipeline.from_profile("ml").audit()
>>> report.lossless
False
>>> report.lossy_steps
('RemoveTashkeel', 'ReduceElongation')
```

--------------------------------

### Normalize Series with Profile and Overrides

Source: https://mhdmartini.github.io/araclean/latest/reference

Normalize each value in a pandas Series using a named profile and optional overrides. Missing values are ignored.

```python
normalize(
    *, profile: str | None = None, **overrides: object
) -> pd.Series[Any]
```
Tokens:
Raw