### Install Model Add-ons (CLI)

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Installs additional dependencies or resources, such as disambiguation rules or lexicons, required by specific language models. This command ensures all necessary components for a model are present.

```bash
# Install addons for Latin LASLA model
pie-extended install-addons lasla

# Output:
# Installing add-ons
# Done
```

--------------------------------

### Install pie-extended via pip

Source: https://github.com/hipster-philology/nlp-pie-taggers/blob/master/README.md

Installs the pie-extended library using pip. Includes an extra index URL for CPU-only PyTorch installations if a GPU is not available.

```bash
pip install pie-extended

```

```bash
pip install pie-extended  --extra-index-url https://download.pytorch.org/whl/cpu

```

--------------------------------

### Install NLP-PIE Taggers Development Version

Source: https://github.com/hipster-philology/nlp-pie-taggers/blob/master/README.md

This bash command installs the NLP-PIE taggers project in development mode. This is intended for developers who are actively modifying the codebase. It requires cloning the repository and setting up a Python environment before execution.

```bash
python setup.py develop

```

--------------------------------

### Format Output with NLP-PIE Taggers

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Demonstrates how to get headers and format annotation lines using the NLP-PIE formatter. It shows the default tab-separated output and how to customize it.

```python
# Get headers
headers = formatter.write_headers()
print(headers)  # "token\tlemma\tPOS\tmorph\r\n"

# Format annotation line
annotation = {"form": "divisa", "lemma": "diuido", "POS": "VER", "morph": "Tense=Perf"}
formatted = formatter.format_line(annotation)
line = formatter.write_line(formatted)
print(line)  # "divisa\tdiuido\tVER\tTense=Perf\r\n"
```

--------------------------------

### Run Taggers from Terminal

Source: https://github.com/hipster-philology/nlp-pie-taggers/blob/master/README.md

Demonstrates how to use the pie-extended command-line interface to download, install addons, and tag text files for a specified language model (e.g., 'lasla').

```bash
pie-extended download lasla
pie-extended install-addons lasla
pie-extended tag lasla your_file.txt

```

--------------------------------

### Example Tagger Output

Source: https://github.com/hipster-philology/nlp-pie-taggers/blob/master/README.md

Illustrates the expected output format when using the Pie Extended tagger on a sample sentence, showing token annotations like 'form', 'lemma', 'POS', and 'morph'.

```json
[{'form': 'lorem', 'lemma': 'lor', 'POS': 'NOMcom', 'morph': 'Case=Acc|Numb=Sing', 'treated': 'lorem'},
 {'form': 'ipsum', 'lemma': 'ipse', 'POS': 'PROdem', 'morph': 'Case=Acc|Numb=Sing', 'treated': 'ipsum'},
 {'form': 'dolor', 'lemma': 'dolor', 'POS': 'NOMcom', 'morph': 'Case=Nom|Numb=Sing', 'treated': 'dolor'},
 {'form': 'sit', 'lemma': 'sum1', 'POS': 'VER', 'morph': 'Numb=Sing|Mood=Sub|Tense=Pres|Voice=Act|Person=3',
  'treated': 'sit'},
 {'form': 'amet', 'lemma': 'amo', 'POS': 'VER', 'morph': 'Numb=Sing|Mood=Sub|Tense=Pres|Voice=Act|Person=3',
  'treated': 'amet'},
 {'form': ',', 'lemma': ',', 'pos': 'PUNC', 'morph': 'MORPH=empty', 'treated': ','},
 {'form': 'consectetur', 'lemma': 'consector2', 'POS': 'VER',
  'morph': 'Numb=Sing|Mood=Sub|Tense=Pres|Voice=Dep|Person=3', 'treated': 'consectetur'},
 {'form': 'adipiscing', 'lemma': 'adipiscor', 'POS': 'VER', 'morph': 'Tense=Pres|Voice=Dep', 'treated': 'adipiscing'},
 {'form': 'elit', 'lemma': 'elio', 'POS': 'VER', 'morph': 'Numb=Sing|Mood=Ind|Tense=Pres|Voice=Act|Person=3',
  'treated': 'elit'},
 {'form': '.', 'lemma': '.', 'pos': 'PUNC', 'morph': 'MORPH=empty', 'treated': '.'}]

```

--------------------------------

### Get Iterator and Processor for Models

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Retrieves the DataIterator and Processor pair required for text processing with a specific model. Each model has unique iterator/processor configurations for tokenization and post-processing. Examples are shown for Latin, Ancient Greek, and Old French.

```python
from pie_extended.cli.utils import get_tagger
from pie_extended.models.lasla.imports import get_iterator_and_processor

# Get tagger and iterator/processor for Latin
tagger = get_tagger("lasla", batch_size=256, device="cpu")
iterator, processor = get_iterator_and_processor(max_tokens=256)

# For Ancient Greek
from pie_extended.models.grc.imports import get_iterator_and_processor as grc_get_iterator
grc_iterator, grc_processor = grc_get_iterator(max_tokens=128)

# For Old French
from pie_extended.models.fro.imports import get_iterator_and_processor as fro_get_iterator
fro_iterator, fro_processor = fro_get_iterator(max_tokens=64)
```

--------------------------------

### Get Iterator and Processor

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Retrieves the DataIterator and Processor pair needed for text processing with a specific model.

```APIDOC
## Get Iterator and Processor

### Description
Returns the DataIterator and Processor pair required for processing text with a specific model. Each model has its own iterator/processor configured with appropriate tokenization and post-processing rules.

### Method
`get_iterator_and_processor(max_tokens)`

### Parameters
#### Path Parameters
None

#### Query Parameters
- **max_tokens** (int) - Optional - The maximum number of tokens to process.

#### Request Body
None

### Request Example
```python
from pie_extended.cli.utils import get_tagger
from pie_extended.models.lasla.imports import get_iterator_and_processor

# Get tagger and iterator/processor for Latin
tagGER = get_tagger("lasla", batch_size=256, device="cpu")
iterator, processor = get_iterator_and_processor(max_tokens=256)

# For Ancient Greek
from pie_extended.models.grc.imports import get_iterator_and_processor as grc_get_iterator
grc_iterator, grc_processor = grc_get_iterator(max_tokens=128)

# For Old French
from pie_extended.models.fro.imports import get_iterator_and_processor as fro_get_iterator
fro_iterator, fro_processor = fro_get_iterator(max_tokens=64)
```

### Response
#### Success Response (200)
- **iterator** (object) - The DataIterator object.
- **processor** (object) - The Processor object.

#### Response Example
(Returns a tuple of iterator and processor objects)
```

--------------------------------

### Get Tagger Instance (Python API)

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Retrieves an ExtensibleTagger instance for a specified language model. This object handles model loading and provides methods for text annotation. It requires downloading model files first if they are not already present.

```python
from pie_extended.cli.utils import get_tagger, download

# Download model files if needed (first time only)
for progress in download("lasla"):
    print(f"Downloaded: {progress}")

# Create tagger with default settings
tagGER = get_tagger("lasla")

```

--------------------------------

### Tag String with ExtensibleTagger.tag_str

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Tags a given string and returns a list of dictionaries, where each dictionary contains annotations for a token (form, lemma, POS, morph). This method requires an initialized tagger, iterator, and processor. The output format is demonstrated with an example.

```python
from pie_extended.cli.utils import get_tagger
from pie_extended.models.lasla.imports import get_iterator_and_processor

# Initialize tagger
tagger = get_tagger("lasla", batch_size=256, device="cpu")

# Tag Latin text
latin_text = "Gallia est omnis divisa in partes tres."
iterator, processor = get_iterator_and_processor()
results = tagger.tag_str(latin_text, iterator=iterator, processor=processor)

# Output: List of annotation dictionaries
# [
#     {'form': 'gallia', 'lemma': 'gallia', 'POS': 'NOMpro', 'morph': 'Case=Nom|Numb=Sing', 'treated': 'gallia'},
#     {'form': 'est', 'lemma': 'sum1', 'POS': 'VER', 'morph': 'Numb=Sing|Mood=Ind|Tense=Pres|Voice=Act|Person=3', 'treated': 'est'},
#     {'form': 'omnis', 'lemma': 'omnis', 'POS': 'ADJqua', 'morph': 'Case=Nom|Numb=Sing', 'treated': 'omnis'},
#     {'form': 'divisa', 'lemma': 'diuido', 'POS': 'VER', 'morph': 'Case=Nom|Numb=Sing|Tense=Perf|Voice=Pass', 'treated': 'diuisa'},
#     {'form': 'in', 'lemma': 'in1', 'POS': 'PRE', 'morph': 'MORPH=empty', 'treated': 'in'},
#     {'form': 'partes', 'lemma': 'pars', 'POS': 'NOMcom', 'morph': 'Case=Acc|Numb=Plur', 'treated': 'partes'},
#     {'form': 'tres', 'lemma': 'tres', 'POS': 'ADJcar', 'morph': 'Case=Acc|Numb=Plur', 'treated': 'tres'},
#     {'form': '.', 'lemma': '.', 'pos': 'PUNC', 'morph': 'MORPH=empty', 'treated': '.'}
# ]

for token in results:
    print(f"{token['form']:15} {token.get('lemma', '_'):15} {token.get('POS', token.get('pos', '_')):10}")

# Output:
# gallia          gallia          NOMpro
# est             sum1            VER
# omnis           omnis           ADJqua
# divisa          diuido          VER
# in              in1             PRE
# partes          pars            NOMcom
# tres            tres            ADJcar
# .               .               PUNC
```

--------------------------------

### Tagger Creation

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Demonstrates how to create a tagger with custom settings and lists available models.

```APIDOC
## Tagger Creation

### Description
Create a tagger with custom settings such as batch size, device, and model path. Lists available models for selection.

### Method
`get_tagger(model, batch_size, device, model_path)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from pie_extended.cli.utils import get_tagger

# Create tagger with custom settings
tagGER = get_tagger(
    model="lasla",
    batch_size=256,      # Larger batches for better GPU utilization
    device="cuda",       # Use GPU (or "cpu" for CPU)
    model_path=None      # Optional: override with custom model file
)

# Available models: "lasla", "grc", "fro", "freem", "fr", "dum", "occ_cont"
greek_tagger = get_tagger("grc", batch_size=128, device="cpu")
french_tagger = get_tagger("fr", batch_size=64)
```

### Response
#### Success Response (200)
Returns a tagger object configured with the specified settings.

#### Response Example
(No direct response example, returns a tagger object)
```

--------------------------------

### List Available Language Models (CLI)

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Lists all available language models supported by Pie Extended, including their language codes and short descriptions. The `--full` option provides more detailed metadata.

```bash
# List available models
pie-extended list

# Output:
# Available Models:
#     - lasla (lat) : LASLA-ENC
#     - fro (fro) : Old French
#     - fr (fra) : Classical French
#     - freem (frm) : Early Modern French
#     - grc (grc) : Ancient Greek
#     - dum (dum) : Old Dutch
#     - occ_cont (occ) : Occitan

# Get full details about each model
pie-extended list --full

# Output includes authors, descriptions, and URLs for each model
```

--------------------------------

### Create Tagger with Custom Settings

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Initializes an NLP-PIE tagger with specified model, batch size, and device. Supports various models like 'lasla', 'grc', 'fro', etc. Batch size can be adjusted for GPU utilization, and 'cuda' or 'cpu' can be selected for the device.

```python
from pie_extended.cli.utils import get_tagger

tagger = get_tagger(
    model="lasla",
    batch_size=256,      # Larger batches for better GPU utilization
    device="cuda",       # Use GPU (or "cpu" for CPU)
    model_path=None      # Optional: override with custom model file
)

# Available models: "lasla", "grc", "fro", "freem", "fr", "dum", "occ_cont"
greek_tagger = get_tagger("grc", batch_size=128, device="cpu")
french_tagger = get_tagger("fr", batch_size=64)
```

--------------------------------

### Download Pre-trained Language Models (CLI)

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Downloads the necessary pre-trained model files for a specified language tagger. Models are saved to a location defined by the PIE_EXTENDED_DOWNLOADS environment variable or the package's default download directory.

```bash
# Download Latin LASLA model
pie-extended download lasla

# Output:
# Starting downloading...
# 12 files to download
# - latin-straight.json downloaded
# - latin-pos.json downloaded
# - latin-needs.json downloaded
# - Mood_Tense_Voice.tar downloaded
# - Gend.tar downloaded
# - Person.tar downloaded
# - Deg.tar downloaded
# - lemma.tar downloaded
# - pos.tar downloaded
# - Numb.tar downloaded
# - Dis.tar downloaded
# - Case.tar downloaded
# Finished !

# Download Ancient Greek model
pie-extended download grc

# Download Old French model
pie-extended download fro
```

--------------------------------

### Configure Iterator and Processor for a Custom NLP-PIE Model

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Shows how to set up a custom tokenizer, processor, and data iterator for a new language model. This includes defining custom normalization logic and exclusion patterns.

```python
# File: pie_extended/models/mymodel/imports.py
from pie_extended.pipeline.iterators.proto import DataIterator, GenericExcludePatterns
from pie_extended.pipeline.postprocessor.proto import ProcessorPrototype
from pie_extended.pipeline.tokenizers.memorizing import MemorizingTokenizer

class MyTokenizer(MemorizingTokenizer):
    def replacer(self, token):
        # Custom normalization logic
        return token.lower()

def get_iterator_and_processor(max_tokens=256):
    tokenizer = MyTokenizer()
    processor = ProcessorPrototype(empty_value="_")
    iterator = DataIterator(
        tokenizer=tokenizer,
        exclude_patterns=[GenericExcludePatterns.Punctuation],
        max_tokens=max_tokens
    )
    return iterator, processor

# Optional: disambiguation support
def addons():
    print("Installing additional resources...")
    return True
```

--------------------------------

### Tag Text Files with Morphological Annotations (CLI)

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Tags one or more text files using a specified language model, generating output files with `.pie` extension containing tab-separated token annotations. Supports various options for device selection, batch size, tokenization, exclusion patterns, and error handling.

```bash
# Tag a single Latin text file
pie-extended tag lasla my_latin_text.txt

# Tag multiple files
pie-extended tag lasla text1.txt text2.txt text3.txt

# Use GPU for faster processing
pie-extended tag lasla my_latin_text.txt --device cuda

# Increase batch size for better throughput
pie-extended tag lasla my_latin_text.txt --batch_size 64

# Use pre-tokenized input (one word per line, blank line = sentence break)
pie-extended tag lasla pretokenized.txt --no-tokenizer

# Add custom exclude patterns for tokens
pie-extended tag lasla my_text.txt --add-pattern "^[0-9]+$" --add-pattern "^\[.*\]$"

# Reset default exclude patterns and add new ones
pie-extended tag lasla my_text.txt --reset-exclude-patterns --add-pattern "^[_]+$"

# Set maximum tokens per sentence
pie-extended tag lasla my_text.txt --max-tokens 128

# Use a custom model file
pie-extended tag lasla my_text.txt --model_path /path/to/custom/model.tar

# Debug mode - shows errors instead of continuing
pie-extended tag lasla my_text.txt --debug

# Allow more failures before stopping
pie-extended tag lasla *.txt --allow-n-failures 10
```

--------------------------------

### Show Model Character Details (CLI)

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Displays information about characters recognized by a specific language model's encoder. This is useful for diagnosing encoding-related issues.

```bash
# Get character details for LASLA model
pie-extended details lasla

# Output: List of characters the model recognizes
```

--------------------------------

### Define Metadata for a Custom NLP-PIE Model

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Illustrates how to define essential metadata for a new custom language model, including its title, language code, authors, description, and a URL for more information.

```python
# File: pie_extended/models/mymodel/__init__.py
from ...utils import Metadata, File, get_path

DESC = Metadata(
    "My Custom Model",           # Title
    "xyz",                       # ISO language code
    ["Your Name"],               # Authors
    "Description of the model",  # Description
    "https://example.com/model"  # URL with more info
)

DOWNLOADS = [
    File("https://example.com/model.tar", "model.tar"),
    File("https://example.com/lexicon.json", "lexicon.json")
]

Models = "<{},lemma,pos>".format(get_path("mymodel", "model.tar"))
```

--------------------------------

### ExtensibleTagger.tag_file

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Tags an entire file and writes the results to a new file in TSV format.

```APIDOC
## ExtensibleTagger.tag_file

### Description
Tags an entire file and writes the results to a new file with a `.pie` extension in TSV format.

### Method
`tagger.tag_file(fpath, iterator, processor, no_tokenizer)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **fpath** (str) - The path to the input file.
- **iterator** (object) - The DataIterator object.
- **processor** (object) - The Processor object.
- **no_tokenizer** (bool) - Set to `True` if the input file is already tokenized.

### Request Example
```python
from pie_extended.cli.utils import get_tagger
from pie_extended.models.lasla.imports import get_iterator_and_processor

# Initialize tagger
tagGER = get_tagger("lasla", batch_size=256, device="cpu")
iterator, processor = get_iterator_and_processor()

# Tag a file - creates 'input.txt.pie' with results
output_path = tagger.tag_file(
    fpath="input.txt",
    iterator=iterator,
    processor=processor,
    no_tokenizer=False  # Set True for pre-tokenized input
)

print(f"Results written to: {output_path}")

# Output file format (TSV):
# token    lemma    POS    morph    treated
# gallia   gallia   NOMpro Case=Nom|Numb=Sing  gallia
# est      sum1     VER    Numb=Sing|Mood=Ind|Tense=Pres|Voice=Act|Person=3  est
# ...    
```

### Response
#### Success Response (200)
- **output_path** (str) - The path to the generated `.pie` file containing the tagged results.

#### Response Example
```
Results written to: input.txt.pie
```
```

--------------------------------

### ExtensibleTagger.tag_str

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Tags a string and returns a list of dictionaries containing token annotations.

```APIDOC
## ExtensibleTagger.tag_str

### Description
Tags a string and returns a list of dictionaries containing token annotations. Each dictionary includes the original form, lemma, part of speech, and morphological features.

### Method
`tagger.tag_str(text, iterator, processor)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **text** (str) - The input string to be tagged.
- **iterator** (object) - The DataIterator object.
- **processor** (object) - The Processor object.

### Request Example
```python
from pie_extended.cli.utils import get_tagger
from pie_extended.models.lasla.imports import get_iterator_and_processor

# Initialize tagger
tagGER = get_tagger("lasla", batch_size=256, device="cpu")

# Tag Latin text
latin_text = "Gallia est omnis divisa in partes tres."
iterator, processor = get_iterator_and_processor()
results = tagger.tag_str(latin_text, iterator=iterator, processor=processor)

# Output: List of annotation dictionaries
# [
#     {'form': 'gallia', 'lemma': 'gallia', 'POS': 'NOMpro', 'morph': 'Case=Nom|Numb=Sing', 'treated': 'gallia'},
#     {'form': 'est', 'lemma': 'sum1', 'POS': 'VER', 'morph': 'Numb=Sing|Mood=Ind|Tense=Pres|Voice=Act|Person=3', 'treated': 'est'},
#     {'form': 'omnis', 'lemma': 'omnis', 'POS': 'ADJqua', 'morph': 'Case=Nom|Numb=Sing', 'treated': 'omnis'},
#     {'form': 'divisa', 'lemma': 'diuido', 'POS': 'VER', 'morph': 'Case=Nom|Numb=Sing|Tense=Perf|Voice=Pass', 'treated': 'diuisa'},
#     {'form': 'in', 'lemma': 'in1', 'POS': 'PRE', 'morph': 'MORPH=empty', 'treated': 'in'},
#     {'form': 'partes', 'lemma': 'pars', 'POS': 'NOMcom', 'morph': 'Case=Acc|Numb=Plur', 'treated': 'partes'},
#     {'form': 'tres', 'lemma': 'tres', 'POS': 'ADJcar', 'morph': 'Case=Acc|Numb=Plur', 'treated': 'tres'},
#     {'form': '.', 'lemma': '.', 'pos': 'PUNC', 'morph': 'MORPH=empty', 'treated': '.'}
# ]

for token in results:
    print(f"{token['form']:15} {token.get('lemma', '_'):15} {token.get('POS', token.get('pos', '_')):10}")

# Output:
# gallia          gallia          NOMpro
# est             sum1            VER
# omnis           omnis           ADJqua
# divisa          diuido          VER
# in              in1             PRE
# partes          pars            NOMcom
# tres            tres            ADJcar
# .               .               PUNC
```

### Response
#### Success Response (200)
- **results** (list) - A list of dictionaries, where each dictionary represents annotations for a token.

#### Response Example
```json
[
    {'form': 'gallia', 'lemma': 'gallia', 'POS': 'NOMpro', 'morph': 'Case=Nom|Numb=Sing', 'treated': 'gallia'},
    {'form': 'est', 'lemma': 'sum1', 'POS': 'VER', 'morph': 'Numb=Sing|Mood=Ind|Tense=Pres|Voice=Act|Person=3', 'treated': 'est'}
]
```
```

--------------------------------

### Define Model Metadata and Downloads in Python

Source: https://github.com/hipster-philology/nlp-pie-taggers/blob/master/README.md

This Python snippet demonstrates how to define metadata, download information, and model configurations for a new NLP-PIE tagger model. It utilizes utility classes like `Metadata` and `File` from `pie_extended.utils` to structure this information. The `Models` variable specifies the filenames and tasks associated with the model.

```python
from pie_extended.utils import Metadata, File, get_path

DESC = Metadata(
    "Foo"
    "language",
    ["Author 1", "Author 2"],
    "A readable description",
    "A link to more information"
)

DOWNLOADS = [
    File("/a/link/to/a/file", "local_name_of_the_file.tar")
]


Models = "<{},task1,task2><{},lemma,pos>".format(
    get_path("foo", "local_name_of_the_file.tar")
)

```

--------------------------------

### Register a Custom Model in NLP-PIE

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Demonstrates the final step in adding a custom model: registering it by including its module name in the `modules` list within the main `__init__.py` file of the `pie_extended.models` package.

```python
# Register model in pie_extended/models/__init__.py
modules = [
    "lasla",
    "fro",
    "fr",
    "freem",
    "grc",
    "dum",
    "occ_cont",
    "mymodel"  # Add your model here
]
```

--------------------------------

### Create a Custom CSV Formatter for NLP-PIE

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Shows how to extend the base Formatter class to create a custom CSV formatter. This allows outputting data in a comma-separated format with quoted fields.

```python
# Custom formatter example
class CSVFormatter(Formatter):
    def write_line(self, formatted):
        return ",".join(f'"{f}"' for f in formatted) + "\n"

    def write_headers(self):
        return self.write_line(self.get_headers())

csv_formatter = CSVFormatter(tasks=["lemma", "POS"])
print(csv_formatter.write_headers())  # '"token","lemma","POS"\n'
```

--------------------------------

### Python API for Tagger Usage

Source: https://github.com/hipster-philology/nlp-pie-taggers/blob/master/README.md

Shows how to use the Pie Extended Python API to load a tagger for a specific model (e.g., 'lasla'), download it if necessary, and process text to retrieve token annotations.

```python
from typing import List
from pie_extended.cli.utils import get_tagger, get_model, download

# In case you need to download
do_download = False
if do_download:
    for dl in download("lasla"):
        x = 1

# model_path allows you to override the model loaded by another .tar
model_name = "lasla"
tagger = get_tagger(model_name, batch_size=256, device="cpu", model_path=None)

sentences: List[str] = ["Lorem ipsum dolor sit amet, consectetur adipiscing elit. "]
# Get the main object from the model (: data iterator + postprocesor
from pie_extended.models.lasla.imports import get_iterator_and_processor
for sentence_group in sentences:
    iterator, processor = get_iterator_and_processor()
    print(tagger.tag_str(sentence_group, iterator=iterator, processor=processor) )

```

--------------------------------

### ExtensibleTagger.iter_tag_token

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

A generator that yields token annotations one at a time, suitable for streaming or custom processing.

```APIDOC
## ExtensibleTagger.iter_tag_token

### Description
Generator that yields token annotations one at a time, useful for streaming large texts or custom processing.

### Method
`tagger.iter_tag_token(text, iterator, processor)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **text** (str) - The input string to be tagged.
- **iterator** (object) - The DataIterator object.
- **processor** (object) - The Processor object.

### Request Example
```python
from pie_extended.cli.utils import get_tagger
from pie_extended.models.lasla.imports import get_iterator_and_processor

tagGER = get_tagger("lasla", batch_size=64, device="cpu")
iterator, processor = get_iterator_and_processor()

latin_text = "Arma virumque cano. Troiae qui primus ab oris."

# Example usage (actual iteration logic would follow)
# for token_annotation in tagger.iter_tag_token(latin_text, iterator=iterator, processor=processor):
#     print(token_annotation)
```

### Response
#### Success Response (200)
- Yields dictionaries, each containing annotations for a single token.

#### Response Example
```json
{
    "form": "arma",
    "lemma": "arma",
    "POS": "NOMcom",
    "morph": "Case=Acc|Numb=Plur",
    "treated": "arma"
}
```
```

--------------------------------

### Tag File with ExtensibleTagger.tag_file

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Tags an entire text file and saves the results to a new file with a `.pie` extension in TSV format. This method requires an initialized tagger, iterator, and processor. The `no_tokenizer` argument can be set to `True` if the input file is already tokenized.

```python
from pie_extended.cli.utils import get_tagger
from pie_extended.models.lasla.imports import get_iterator_and_processor

# Initialize tagger
tagger = get_tagger("lasla", batch_size=256, device="cpu")
iterator, processor = get_iterator_and_processor()

# Tag a file - creates 'input.txt.pie' with results
output_path = tagger.tag_file(
    fpath="input.txt",
    iterator=iterator,
    processor=processor,
    no_tokenizer=False  # Set True for pre-tokenized input
)

print(f"Results written to: {output_path}")

# Output file format (TSV):
# token    lemma    POS    morph    treated
# gallia   gallia   NOMpro Case=Nom|Numb=Sing  gallia
# est      sum1     VER    Numb=Sing|Mood=Ind|Tense=Pres|Voice=Act|Person=3  est
# ...
```

--------------------------------

### Formatter: Output Formatting

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Handles the formatting of tagged results. By default, it produces output in TSV (Tab-Separated Values) format. It can be configured to format specific tasks.

```python
from pie_extended.pipeline.formatters.proto import Formatter

# Create formatter for specific tasks
formatter = Formatter(tasks=["lemma", "POS", "morph"])
```

--------------------------------

### MemorizingTokenizer: Preserve Original Text Forms

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

A tokenizer that remembers original input forms, allowing restoration of text after normalization. It can be customized to perform specific replacements during tokenization. It also supports bypassing the tokenizer for pre-tokenized input and can be reset between documents.

```python
from pie_extended.pipeline.tokenizers.memorizing import MemorizingTokenizer

class CustomTokenizer(MemorizingTokenizer):
    def replacer(self, token):
        # Normalize 'v' to 'u' for Latin
        return token.replace('v', 'u').replace('V', 'U')

tokenizer = CustomTokenizer()

# Tokenize and memorize
text = "Gallia est omnis divisa in partes tres."
for sentence in tokenizer.sentence_tokenizer(text):
    print(f"Normalized: {sentence}")

# Access memory of original vs normalized forms
for idx, original, normalized in tokenizer.tokens:
    if original != normalized:
        print(f"Token {idx}: '{original}' -> '{normalized}'")

# Reset between documents
tokenizer.reset()

# Bypass tokenizer for pre-tokenized input
pretokenized = "word1\nword2\nword3\n\nsentence2word1\nsentence2word2"
for sentence in tokenizer.bypass_tokenizer(pretokenized):
    print(f"Sentence: {sentence}")
```

--------------------------------

### ProcessorPrototype: Post-processing Tagged Output

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Serves as a base class for post-processing tagged output. It handles task mapping and reinsertion of excluded tokens. Custom processors can inherit from this to modify tagged data, such as converting lemmas to uppercase.

```python
from pie_extended.pipeline.postprocessor.proto import ProcessorPrototype, ChainedProcessor

# Basic processor
processor = ProcessorPrototype(empty_value="_")
processor.set_tasks(["lemma", "pos", "morph"])

# Get dictionary representation of a tagged token
result = processor.get_dict("cano", ["cano", "VER", "Mood=Ind"])
print(result)
# [{'form': 'cano', 'lemma': 'cano', 'pos': 'VER', 'morph': 'Mood=Ind'}]

# Reinsert excluded tokens (e.g., punctuation)
punct = processor.reinsert(",")
print(punct)
# {'form': ',', 'lemma': '_', 'pos': '_', 'morph': '_'}

# Create a chained processor for custom post-processing
class UppercaseLemmaProcessor(ChainedProcessor):
    def get_dict(self, token, tags):
        results = self.head_processor.get_dict(token, tags)
        for result in results:
            if 'lemma' in result:
                result['lemma'] = result['lemma'].upper()
        return results

processor = UppercaseLemmaProcessor(ProcessorPrototype(empty_value="_"))
processor.set_tasks(["lemma", "pos"])
result = processor.get_dict("cano", ["cano", "VER"])
print(result)
# [{'form': 'cano', 'lemma': 'CANO', 'pos': 'VER'}]
```

--------------------------------

### Iterate and Tag Tokens with ExtensibleTagger.iter_tag_token

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Provides a generator that yields token annotations one by one. This is useful for processing large texts or implementing custom annotation workflows. It requires an initialized tagger, iterator, and processor.

```python
from pie_extended.cli.utils import get_tagger
from pie_extended.models.lasla.imports import get_iterator_and_processor

tagger = get_tagger("lasla", batch_size=64, device="cpu")
iterator, processor = get_iterator_and_processor()

latin_text = "Arma virumque cano. Troiae qui primus ab oris."
```

--------------------------------

### Stream Tokens with Sentence Breaks

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Iterates through tokens, yielding None for sentence breaks. This allows for processing text sentence by sentence. The `empty_token_on_sent_break` parameter controls the behavior at sentence boundaries.

```python
for token in tagger.iter_tag_token(
    data=latin_text,
    iterator=iterator,
    processor=processor,
    no_tokenizer=False,
    empty_token_on_sent_break=True  # Yields None between sentences
):
    if token is None:
        print("--- sentence break ---")
    else:
        print(f"{token['form']} -> {token.get('lemma', '_')}")
```

--------------------------------

### DataIterator: Tokenization and Exclusion

Source: https://context7.com/hipster-philology/nlp-pie-taggers/llms.txt

Handles text tokenization and allows for custom token exclusion patterns. It can be configured with predefined or custom regular expressions to filter out unwanted tokens like punctuation or specific markers. The iterator also tracks removed tokens.

```python
from pie_extended.pipeline.iterators.proto import DataIterator, GenericExcludePatterns
from pie_extended.pipeline.tokenizers.simple_tokenizer import SimpleTokenizer

# Create iterator with default tokenizer
iterator = DataIterator(
    tokenizer=SimpleTokenizer(),
    max_tokens=256
)

# Create iterator with exclude patterns
iterator = DataIterator(
    tokenizer=SimpleTokenizer(),
    exclude_patterns=[
        GenericExcludePatterns.Punctuation_and_Underscore,  # Excludes punctuation
        GenericExcludePatterns.PassageMarker,               # Excludes _Passage_* markers
        r"^[0-9]+$"                                          # Custom: exclude numbers
    ],
    max_tokens=128
)

# Process text
text = "Hello, world! This is a test. _Passage_123"
for sentence, length, removed in iterator(text, lower=False):
    print(f"Tokens: {sentence}")
    print(f"Removed: {removed}")

# Exclude tokens and track what was removed
sentence = ["Je", "suis", "content", ",", "mais", "...", "fatigué", "."]
clean, removed = iterator.exclude_tokens(sentence)
print(f"Clean: {clean}")      # ['Je', 'suis', 'content', 'mais', 'fatigué']
print(f"Removed: {removed}")  # {3: ',', 5: '...', 7: '.'}

# Add patterns dynamically
iterator.add_pattern(r"^[.*]$")  # Exclude bracketed text

# Reset all patterns
iterator.reset_patterns()
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.