### Make Setup Script Executable and Run Interactive Setup

Source: https://github.com/amir-zeldes/hebpipe/blob/master/docker/README.md

These bash commands are used to prepare and launch the interactive setup script for the HebPipe Docker container. The first command grants execute permissions to the script, and the second command executes it, initiating the configuration process for the Docker build.

```bash
chmod +x run_hebpipe_docker.sh
./run_hebpipe_docker.sh
```

--------------------------------

### Clone HebPipe Repository and Navigate to Docker Directory

Source: https://github.com/amir-zeldes/hebpipe/blob/master/docker/README.md

This snippet shows the necessary bash commands to clone the HebPipe repository from GitHub and then change the current directory to the 'docker' folder within the cloned repository. These are the initial steps for setting up the Docker environment.

```bash
git clone https://github.com/amir-zeldes/HebPipe.git
cd HebPipe
cd docker
```

--------------------------------

### Hebrew NLP Docker Deployment - CPU and GPU

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Illustrates how to deploy HebPipe using Docker containers for both CPU and GPU acceleration. It includes commands for setting up CPU and GPU environments via `docker-compose`, processing files within the container, and mounting local directories for batch operations. The example also shows a snippet of a Dockerfile configuration.

```bash
# CPU-based deployment
cd docker
docker-compose -f docker-compose-cpu.yml up -d

# GPU-based deployment (requires nvidia-docker)
docker-compose -f docker-compose-gpu.yml up -d

# Process files in container
docker exec -it hebpipe python -m hebpipe /data/input.txt

# Mount local directory for batch processing
docker run -v /local/data:/data hebpipe python -m hebpipe /data/*.txt

# Example Dockerfile configuration (docker-compose-cpu.yml):
# services:
#   hebpipe:
#     image: python:3.10
#     volumes:
#       - ./data:/data
#     command: python -m hebpipe -wtpldec /data/*.txt
#     working_dir: /app

# GPU version uses nvidia/cuda base image for acceleration
# CPU version suitable for lightweight deployments
```

--------------------------------

### Hebrew Sentence Tokenization Example

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Demonstrates basic sentence tokenization using the HebPipe library. It iterates through predefined sentences and prints them with their respective indices. This showcases the library's ability to handle basic punctuation and sentence structures in Hebrew.

```python
sentences = [
    ['עיפרון', 'הוא', 'כלי', 'כתיבה', '.'],
    ['ילדים', 'משחקים', 'בגן', '.']
]

for i, sent in enumerate(sentences):
    print(f"Sentence {i+1}: {' '.join(sent)}")
```

--------------------------------

### Access HebPipe Docker Container (CPU and GPU)

Source: https://github.com/amir-zeldes/hebpipe/blob/master/docker/README.md

This snippet demonstrates how to access the running HebPipe Docker container using the 'docker exec' command. It provides separate commands for accessing the CPU version ('hebpipe-cpu') and the GPU version ('hebpipe-gpu'), allowing users to interact with the pipeline inside the container.

```bash
# For CPU version
docker exec -it hebpipe-cpu bash

# For GPU version
docker exec -it hebpipe-gpu bash
```

--------------------------------

### Stop HebPipe Docker Compose Services

Source: https://github.com/amir-zeldes/hebpipe/blob/master/docker/README.md

This bash command is used to gracefully stop and remove the containers, networks, and volumes defined in the 'docker-compose.yml' file. It's the standard way to shut down the HebPipe Docker environment when it's no longer needed.

```bash
docker compose down
```

--------------------------------

### Run HebPipe NLP Pipeline within Docker Container

Source: https://github.com/amir-zeldes/hebpipe/blob/master/docker/README.md

This Python command is executed inside the HebPipe Docker container to process a text file. It uses the '-m hebpipe' flag to run the module and specifies the path to the input file. The output is typically saved in the same directory as the input file.

```bash
python -m hebpipe /path/to/your/file.txt
```

--------------------------------

### Preloaded Models - Batch Processing (Python)

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Optimizes performance for processing multiple documents by preloading NLP models once and reusing them. This involves initializing tokenizers, taggers, parsers, and lemmatizers before calling the `nlp()` function. The `preloaded` argument is used to pass these initialized models, reducing overhead for batch tasks.

```python
from hebpipe.heb_pipe import nlp, init_lemmatizer
from rftokenizer import RFTokenizer
from hebpipe.lib.xrenner import Xrenner
from diaparser.parsers import Parser
import os

# Set model directory
model_dir = "./hebpipe/models/"

# Preload all models once
rf_tok = RFTokenizer(model=model_dir + "heb.sm3")
xrenner = Xrenner(model=model_dir + "heb.xrm")
parser = Parser.load(model_dir + "heb.diaparser", verbose=False)
lemmatizer = init_lemmatizer(cpu=False)

# Process multiple texts efficiently
texts = [
    "עיפרון הוא כלי כתיבה",
    "בית ספר גדול בעיר",
    "ילדים משחקים בגן"
]

preloaded = (rf_tok, xrenner, parser, lemmatizer)

for text in texts:
    output = nlp(
        text,
        do_whitespace=True,
        do_tok=True,
        do_tag=True,
        do_lemma=True,
        do_parse=True,
        do_entity=True,
        preloaded=preloaded
    )
    print(output)

# Models remain in memory across iterations for efficiency
```

--------------------------------

### Command Line Processing - Partial Pipeline

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Execute specific NLP steps of the HebPipe pipeline from the command line, useful for pre-processed or specific analysis needs.

```APIDOC
## Command Line Processing - Partial Pipeline

### Description
Execute specific NLP steps of the HebPipe pipeline from the command line, useful for pre-processed or specific analysis needs.

### Method
`python -m hebpipe`

### Endpoint
N/A (Command Line Tool)

### Parameters
#### Command Line Arguments
- **`example_in.txt`** (file) - Input text file.
- **`-s sent`** (flag) - Specifies that the input is pre-tokenized with sentence boundaries.
- **`-pld`** (flags) - Enable POS tagging, lemmatization, and dependency parsing.
- **`-d`** (flag) - Enable dependency parsing.
- **`-wt`** (flag) - Enable whitespace tokenization.
- **`-o pipes`** (flag) - Specify output format as pipe-separated tokens.

### Request Example
```bash
# POS tag and parse pre-tokenized file with XML sentence boundaries
python -m hebpipe -pld -s sent example_in.txt

# Parse tagged TreeTagger SGML file into CoNLL format
python -m hebpipe -d -s sent example_in.tt

# Tokenize only with pipe separators
python -m hebpipe -wt -o pipes example_in.txt
```

### Response
#### Success Response (200)
- Output format varies based on the specified options (e.g., CoNLL-U, pipes).

#### Response Example (Pipes format)
```
עיפרון
הוא
כלי
כתיבה
ידני
```
```

--------------------------------

### Command Line Processing - Selective Pipeline

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Allows selective processing of Hebrew text by enabling specific NLP steps via the command line. This is useful for pre-tokenized text or when only certain analyses are required. Input can be text files, and output formats vary (e.g., CoNLL-U, pipes).

```bash
# POS tag and parse pre-tokenized file with XML sentence boundaries
python -m hebpipe -pld -s sent example_in.txt

# Parse tagged TreeTagger SGML file into CoNLL format
python -m hebpipe -d -s sent example_in.tt

# Tokenize only with pipe separators
python -m hebpipe -wt -o pipes example_in.txt

# Output example for pipes format:
# עיפרון
# הוא
# כלי
# כתיבה
# ידני
```

--------------------------------

### Command Line Processing - Full Pipeline

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Process Hebrew text files through the complete NLP pipeline using the command line. Supports various options for specifying steps and output directories.

```APIDOC
## Command Line Processing - Full Pipeline

### Description
Process Hebrew text files through the complete NLP pipeline using the command line. Supports various options for specifying steps and output directories.

### Method
`python -m hebpipe`

### Endpoint
N/A (Command Line Tool)

### Parameters
#### Command Line Arguments
- **`example_in.txt`** (file) - Input text file.
- **`--dirout`** (directory) - Optional: Specifies the output directory for processed files.
- **`-wtpldec`** (flags) - Optional: Explicitly enable whitespace tokenization, POS tagging, lemmatization, dependency parsing, and entity recognition.

### Request Example
```bash
# Process a file with all NLP steps (default behavior)
python -m hebpipe example_in.txt

# Process with explicit options for all steps
python -m hebpipe -wtpldec example_in.txt

# Process multiple files with output to specific directory
python -m hebpipe -wtpldec --dirout /output/directory/ *.txt
```

### Response
#### Success Response (200)
- Output is typically in CoNLL-U format, detailing token information, POS tags, lemmas, and dependency relations.

#### Response Example (CoNLL-U format)
```
1	עיפרון	עיפרון	NOUN	NOUN	Gender=Masc|Number=Sing	2	nsubj	_	_
2	הוא
הוא	PRON	PRON	Gender=Masc|Number=Sing|Person=3	0	root	_	_
3	כלי	כלי	NOUN	NOUN	Gender=Masc|Number=Sing	2	nmod	_	_
4	כתיבה	כתיבה	NOUN	NOUN	Gender=Fem|Number=Sing	3	nmod	_	_
5
ידני	ידני	ADJ	ADJ	Gender=Masc|Number=Sing	3	amod	_	_
```
```

--------------------------------

### Command Line Processing - Full Pipeline

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Processes Hebrew text files using the full HebPipe NLP pipeline via the command line. It supports various options for enabling specific analysis steps and directing output. Input is typically a text file, and output is in CoNLL-U format.

```bash
# Process a file with all NLP steps (default behavior)
python -m hebpipe example_in.txt

# Process with explicit options for all steps
python -m hebpipe -wtpldec example_in.txt

# Process multiple files with output to specific directory
python -m hebpipe -wtpldec --dirout /output/directory/ *.txt

# Example input file (example_in.txt):
# עיפרון הוא כלי כתיבה ידני

# Expected output (CoNLL-U format):
# 1	עיפרון	עיפרון	NOUN	NOUN	Gender=Masc|Number=Sing	2	nsubj	_	_
# 2
הוא
הוא
PRON	PRON	Gender=Masc|Number=Sing|Person=3	0	root	_	_
# 3	כלי	כלי	NOUN	NOUN	Gender=Masc|Number=Sing	2	nmod	_	_
# 4	כתיבה	כתיבה	NOUN	NOUN	Gender=Fem|Number=Sing	3	nmod	_	_
# 5	ידני	ידני	ADJ	ADJ	Gender=Masc|Number=Sing	3	amod	_	_
```

--------------------------------

### Preloaded Models - Batch Processing

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Optimize performance for processing multiple documents by preloading models once and reusing them across batch processing using the `nlp()` function with the `preloaded` argument.

```APIDOC
## Preloaded Models - Batch Processing

### Description
Optimize performance for processing multiple documents by preloading models once and reusing them across batch processing using the `nlp()` function with the `preloaded` argument.

### Method
`hebpipe.heb_pipe.nlp()` with `preloaded` models.

### Endpoint
N/A (Python Function)

### Parameters
#### Setup
- **`model_dir`** (str) - Path to the directory containing HebPipe models.
- **`RFTokenizer`**, **`Xrenner`**, **`Parser`**, **`init_lemmatizer`** - Functions/classes to load specific NLP models.

#### Function Arguments for `nlp()`
- **`text`** (str) - The Hebrew text to process.
- **`preloaded`** (tuple) - A tuple containing already loaded model instances (RFTokenizer, Xrenner, Parser, lemmatizer).
- Other `nlp()` arguments (e.g., `do_whitespace`, `do_tok`, etc.) apply as usual.

### Request Example
```python
from hebpipe.heb_pipe import nlp, init_lemmatizer
from rftokenizer import RFTokenizer
from hebpipe.lib.xrenner import Xrenner
from diaparser.parsers import Parser
import os

# Set model directory
model_dir = "./hebpipe/models/"

# Preload all models once
rf_tok = RFTokenizer(model=model_dir + "heb.sm3")
xrenner = Xrenner(model=model_dir + "heb.xrm")
parser = Parser.load(model_dir + "heb.diaparser", verbose=False)
lemmatizer = init_lemmatizer(cpu=False)

# Process multiple texts efficiently
texts = [
    "עיפרון הוא כלי כתיבה",
    "בית ספר גדול בעיר",
    "ילדים משחקים בגן"
]

preloaded_models = (rf_tok, xrenner, parser, lemmatizer)

for text in texts:
    output = nlp(
        text,
        do_whitespace=True,
        do_tok=True,
        do_tag=True,
        do_lemma=True,
        do_parse=True,
        do_entity=True,
        preloaded=preloaded_models
    )
    print(output)

# Models remain in memory across iterations for efficiency
```

### Response
#### Success Response (200)
- The `nlp()` function returns processed text for each input string in the specified format.

#### Response Example (Output for one text in the batch)
```
1	עיפרון	עיפרון	NOUN	NOUN	Gender=Masc|Number=Sing	2	nsubj	_	_
2
הוא	הוא	PRON	PRON	Gender=Masc|Number=Sing|Person=3	0	root	_	_
3	כלי	כלי	NOUN	NOUN	Gender=Masc|Number=Sing	2	nmod	_	_
4	כתיבה	כתיבה	NOUN	NOUN	Gender=Fem|Number=Sing	3	nmod	_	_
```
```

--------------------------------

### Hebrew NLP Model Management - Automatic Download

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Provides Python code to automatically check for and download necessary NLP models for HebPipe. It uses `check_requirements` to verify existing models and `download_requirements` to fetch missing ones. The process ensures all dependencies like tokenizers, entity recognition, and lemmatization models are available.

```python
from hebpipe.heb_pipe import check_requirements, download_requirements

# Verify all model files exist
models_ok = check_requirements()

# Returns False if any models missing:
# - heb.sm3 (Random Forest tokenizer)
# - heb.xrm (Entity recognition model)
# - heb.diaparser (Dependency parser)
# - stanza/he_lemmatizer.pt (Lemmatization model)
# - stanza/he_htb.pretrain.pt (Pre-trained embeddings)

if not models_ok:
    print("Missing models detected")
    # Download from gucorpling.org
    download_requirements(models_ok=False)
    print("Models downloaded successfully")
else:
    print("All models present")

# Models are downloaded to ./hebpipe/models/ directory
# Total size: ~2-3 GB for all models
# Download performed once, then cached locally
```

--------------------------------

### Whitespace Tokenization - Super-token Segmentation (Python)

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Performs initial tokenization by splitting text into whitespace-delimited super-tokens, preserving sentence structure. It handles Hebrew abbreviations and punctuation, and can automatically detect sentence boundaries. The output maintains original spacing for reconstruction.

```python
from hebpipe.lib.whitespace_tokenize import tokenize as whitespace_tokenize

# Tokenize with automatic sentence detection
text = "עיפרון הוא כלי כתיבה. ילדים משחקים."
abbr_file = "./hebpipe/data/heb_abbr.tab"

tokenized = whitespace_tokenize(
    text,
    abbr=abbr_file,
    add_sents=True,
    from_pipes=False
)

# Returns one super-token per line with <s> tags for sentences:
# <s>
# עיפרון
# הוא
# כלי
# כתיבה
# </s>
# <s>
# ילדים
# משחקים
# </s>

# Handles Hebrew-specific abbreviations and punctuation
# Preserves original spacing for reconstruction
```

--------------------------------

### Dependency Parsing - Syntactic Analysis (Python)

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Parses morphologically analyzed Hebrew text to generate dependency trees, including head relations and syntactic functions. It utilizes a loaded Diaparser model and takes CoNLL-U input with POS tags and lemmas. The output provides head indices and dependency relations like nsubj, root, and nmod.

```python
from hebpipe.heb_pipe import diaparse
from diaparser.parsers import Parser

# Load parser model
parser = Parser.load("./hebpipe/models/heb.diaparser", verbose=False)

# CoNLL-U input with POS tags and lemmas
conllu_input = """
1\tעיפרון\tעיפרון\tNOUN\tNOUN\tGender=Masc|Number=Sing\t_\t_\t_\t_
2\tהוא\tהוא\tPRON\tPRON\tGender=Masc|Number=Sing|Person=3\t_\t_\t_\t_
3\tכלי\tכלי\tNOUN\tNOUN\tGender=Masc|Number=Sing\t_\t_\t_\t_

"""

# Parse to generate dependency relations
parsed_output = diaparse(parser, conllu_input)

# Output includes head indices and dependency relations:
# 1	עיפרון	עיפרון	NOUN	NOUN	Gender=Masc|Number=Sing	2	nsubj	_	_
# 2	הוא	הוא	PRON	PRON	Gender=Masc|Number=Sing|Person=3	0	root	_	_
# 3	כלי	כלי	NOUN	NOUN	Gender=Masc|Number=Sing	2	nmod	_	_

# Dependency types include: nsubj, root, nmod, amod, det, case, etc.
```

--------------------------------

### Sentence Splitting - Automatic Segmentation (Python)

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Splits tokenized Hebrew text into sentences using linguistic heuristics and newline detection. The input is a list of tokens, and the output is a list of lists, where each inner list represents a sentence. This is a fundamental step for downstream sentence-level analysis.

```python
from hebpipe.lib.sent_split import toks_to_sents

# Input: list of tokens (one per line or as list)
tokens = [
    "עיפרון",
    "הוא",
    "כלי",
    "כתיבה",
    ".",
    "ילדים",
    "משחקים",
    "בגן",
    "."
]

# Split into sentence lists
sentences = toks_to_sents(tokens)

# Returns list of sentence lists:
# [

```

--------------------------------

### Lemmatization - Morphological Analysis (Python)

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Performs lemmatization on Hebrew text, extracting morphological features and handling specific patterns like binyan forms and plurals. It takes CoNLL-U formatted input and a list of morphological features to produce a list of lemmas. Normalization of final letters and lexicon lookup are included.

```python
from hebpipe.heb_pipe import lemmatize, init_lemmatizer

# Initialize lemmatizer
lemmatizer = init_lemmatizer(cpu=False, no_post_process=False)

# Input CoNLL-U format with morphological features
conllu_input = """
1\tעיפרונות\t_\tNOUN\tNOUN\t_\t_\t_\t_\t_
2\tכחולים\t_\tADJ\tADJ\t_\t_\t_\t_\t_
"""

morphs = ["Gender=Masc|Number=Plur", "Gender=Masc|Number=Plur"]

# Lemmatize with morphological awareness
lemmas = lemmatize(lemmatizer, conllu_input, morphs)

# Returns list: ['עיפרון', 'כחול']
# Automatically handles:
# - Plural to singular conversion
# - Final letter normalization (ם->מ, ן->נ, ך->כ, ץ->צ, ף->פ)
# - Verbal binyan patterns
# - Lexicon lookup for ambiguous cases

for i, lemma in enumerate(lemmas):
    print(f"Word: {morphs[i]} -> Lemma: {lemma}")
```

--------------------------------

### Programmatic API - nlp() Function

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Integrate HebPipe's core NLP processing into Python applications using the `nlp()` function. Supports various processing options and output formats.

```APIDOC
## Programmatic API - nlp() Function

### Description
Integrate HebPipe's core NLP processing into Python applications using the `nlp()` function. Supports various processing options and output formats like CoNLL-U or pipes.

### Method
`hebpipe.heb_pipe.nlp()`

### Endpoint
N/A (Python Function)

### Parameters
#### Function Arguments
- **`input_text`** (str) - The Hebrew text to process.
- **`do_whitespace`** (bool) - Enable whitespace tokenization (default: True).
- **`do_tok`** (bool) - Enable tokenization (default: True).
- **`do_tag`** (bool) - Enable Part-of-Speech tagging (default: True).
- **`do_lemma`** (bool) - Enable lemmatization (default: True).
- **`do_parse`** (bool) - Enable dependency parsing (default: True).
- **`do_entity`** (bool) - Enable named entity recognition (default: True).
- **`out_mode`** (str) - Output format ('conllu' or 'pipes', default: 'conllu').
- **`sent_tag`** (str) - Sentence boundary tagging ('none', 'xml', 'both', default: 'none').
- **`from_pipes`** (bool) - If True, input is pre-tokenized with pipes (default: False).
- **`preloaded`** (tuple) - A tuple of preloaded models for optimized batch processing.

### Request Example
```python
from hebpipe.heb_pipe import nlp

input_text = "עיפרון הוא כלי כתיבה ידני"

# Basic usage - full processing with CoNLL-U output
output = nlp(
    input_text,
    do_whitespace=True,
    do_tok=True,
    do_tag=True,
    do_lemma=True,
    do_parse=True,
    do_entity=True,
    out_mode="conllu",
    sent_tag="both"
)

# Minimal processing - tokenization only with pipes output
output_pipes = nlp(
    input_text,
    do_whitespace=True,
    do_tok=True,
    do_tag=False,
    do_lemma=False,
    do_parse=False,
    do_entity=False,
    out_mode="pipes"
)

# Pre-tokenized input with pipes
pretok_input = "ה|עיפרון\nהוא\nכלי|כתיבה"
output = nlp(
    pretok_input,
    do_whitespace=False,
    from_pipes=True,
    out_mode="conllu"
)

print(output)
```

### Response
#### Success Response (200)
- Returns a string containing the processed text in the specified `out_mode` format (CoNLL-U or pipes).

#### Response Example (CoNLL-U format from basic usage)
```
1	עיפרון	עיפרון	NOUN	NOUN	Gender=Masc|Number=Sing	2	nsubj	_	_
2	הוא	הוא	PRON	PRON	Gender=Masc|Number=Sing|Person=3	0	root	_	_
3	כלי	כלי	NOUN	NOUN	Gender=Masc|Number=Sing	2	nmod	_	_
4	כתיבה	כתיבה	NOUN	NOUN	Gender=Fem|Number=Sing	3	nmod	_	_
5
ידני	ידני	ADJ	ADJ	Gender=Masc|Number=Sing	3	amod	_	_
```
```

--------------------------------

### Entity Recognition - Named Entity Detection (Python)

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Identifies and classifies named entity spans within Hebrew text using the xrenner system. It takes CoNLL-U parsed input and annotates entities with types like person, place, and organization. The output is in CoNLL format with entity annotations and includes coreference chains.

```python
from hebpipe.lib.xrenner import Xrenner

# Initialize entity recognizer
xrenner = Xrenner(model="./hebpipe/models/heb.xrm")

# Input: parsed CoNLL-U format
conllu_parsed = """
# sent_id = 1
1	ירושלים	ירושלים	PROPN	PROPN	_	3	nsubj	_	Entity=(place-1
2	היא	הוא	PRON	PRON	_	3	cop	_	_
3	עיר	עיר	NOUN	NOUN	_	0	root	_	_
4	גדולה	גדול	ADJ	ADJ	_	3	amod	_	_

"""

# Analyze entities
xrenner.docname = "example"
output = xrenner.analyze(conllu_parsed, "conll_sent")

# Returns CoNLL format with entity annotations in final column:
# 1	ירושלים	ירושלים	PROPN	PROPN	_	3	nsubj	_	Entity=(place-1)
# 2	היא	הוא	PRON	PRON	_	3	cop	_	Coref=(place-1)
# 3	עיר	עיר	NOUN	NOUN	_	0	root	_	_
# 4	גדולה	גדול	ADJ	ADJ	_	3	amod	_	_

# Entity types: person, place, organization, time, abstract, object
# Includes coreference chains linking mentions
```

--------------------------------

### Programmatic API - nlp() Function (Python)

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Provides the core NLP processing function `nlp()` for integration into Python applications. It accepts input text and various boolean flags to control which NLP steps are performed. The output can be formatted as CoNLL-U or SGML strings, with options for sentence tagging.

```python
from hebpipe.heb_pipe import nlp

# Basic usage - full processing
input_text = "עיפרון הוא כלי כתיבה ידני"
output = nlp(
    input_text,
    do_whitespace=True,
    do_tok=True,
    do_tag=True,
    do_lemma=True,
    do_parse=True,
    do_entity=True,
    out_mode="conllu",
    sent_tag="both"
)

# Minimal processing - tokenization only
output_pipes = nlp(
    input_text,
    do_whitespace=True,
    do_tok=True,
    do_tag=False,
    do_lemma=False,
    do_parse=False,
    do_entity=False,
    out_mode="pipes"
)

# Pre-tokenized input with pipes
pretok_input = "ה|עיפרון\nהוא\nכלי|כתיבה"
output = nlp(
    pretok_input,
    do_whitespace=False,
    from_pipes=True,
    out_mode="conllu"
)

# Result includes: token ID, form, lemma, UPOS, XPOS, features, head, deprel
# Output is returned as string in CoNLL-U format
```

--------------------------------

### Morphological Tokenization - Sub-token Segmentation (Python)

Source: https://context7.com/amir-zeldes/hebpipe/llms.txt

Segments Hebrew super-tokens into finer morphological units using a Random Forest classifier. The input is a list of super-tokens, and the output is a list where each token is segmented by pipes (|) indicating morpheme boundaries. This allows for detailed analysis of word structures.

```python
from rftokenizer import RFTokenizer

# Initialize tokenizer with Hebrew model
rf_tok = RFTokenizer(model="./hebpipe/models/heb.sm3")

# Input: one super-token per line
super_tokens = ["בבית", "והעיפרון", "שלנו"]

# Segment into morphemes with pipe separators
segmented = rf_tok.rf_tokenize(super_tokens)

# Returns:
# ['ב|בית', 'ו|ה|עיפרון', 'של|נו']

# Each pipe indicates morpheme boundary:
# ב|בית -> [preposition ב, noun בית]
# ו|ה|עיפרון -> [conjunction ו, definite article ה, noun עיפרון]
# של|נו -> [possessive של, pronoun נו]

for token in segmented:
    morphemes = token.split('|')
    print(f"{token} has {len(morphemes)} morphemes: {morphemes}")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.