### Install wtpsplit

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Install the library via pip with optional support for ONNX acceleration.

```bash
# Base installation
pip install wtpsplit

# With ONNX GPU support
pip install wtpsplit[onnx-gpu]

# With ONNX CPU support
pip install wtpsplit[onnx-cpu]
```

--------------------------------

### Install wtpsplit

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Install the wtpsplit library using pip. For ONNX support, choose either the GPU or CPU version.

```bash
pip install wtpsplit
```

```bash
pip install wtpsplit[onnx-gpu]
```

```bash
pip install wtpsplit[onnx-cpu]
```

--------------------------------

### Clone Repository and Install Requirements

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Clones the wtpsplit repository and installs the necessary Python packages, including adapters, to prepare for LoRA adaptation.

```bash
git clone https://github.com/segment-any-text/wtpsplit
cd wtpsplit
pip install -r requirements.txt
pip install adapters==0.2.1 --no-dependencies
cd ..
```

--------------------------------

### Initialize SaT with ONNX Runtime Providers

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Initialize the SaT model with specified ONNX Runtime providers for accelerated inference. This example shows how to enable both CUDA and CPU execution.

```python
sat = SaT("sat-3l-sm", ort_providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
```

--------------------------------

### Load LoRA Modules for Domain/Style Adaptation (Universal Dependencies)

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Load LoRA modules for domain and style adaptation. Requires both lang_code and style_or_domain. This example loads UD for English.

```python
# requires both lang_code and style_or_domain
# for available ones, check the <model_repository>/loras folder
sat_lora = SaT("sat-3l", style_or_domain="ud", language="en")
sat_lora.split("Hello this is a test But this is different now Now the next one starts looool")
```

--------------------------------

### Load LoRA Modules for Domain/Style Adaptation (Code-Switching)

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Load LoRA modules for domain and style adaptation. This example loads a code-switching module for Spanish-English.

```python
# now for a highly distinct domain
sat_lora_distinct = SaT("sat-12l", style_or_domain="code-switching", language="es-en")
sat_lora_distinct.split("in the morning over there cada vez que yo decía algo él me decía algo")
```

--------------------------------

### Language-Aware Gaussian Prior Defaults

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Illustrates how to use language-specific defaults for Gaussian prior parameters like target length and spread by providing a 'lang_code'. Examples for Chinese and German are shown.

```python
# Uses Chinese defaults: target_length=45, spread=15
sat.split(text, max_length=100, prior_type="gaussian", 
          prior_kwargs={"lang_code": "zh"})

# Uses German defaults: target_length=90, spread=35
sat.split(text, max_length=150, prior_type="gaussian", 
          prior_kwargs={"lang_code": "de"})
```

--------------------------------

### Get default punctuation threshold for a style

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Retrieves the default punctuation threshold for a specified language and style.

```python
wtp.get_threshold("en", "ud", return_punctuation_threshold=True)
```

--------------------------------

### Get Sentence Boundary Probabilities

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Obtain raw probability predictions for sentence boundaries at each character position using `predict_proba`. This allows for custom processing of boundary detection. Batch prediction is supported for efficiency.

```python
from wtpsplit import SaT
import numpy as np

sat = SaT("sat-3l-sm")

text = "Hello world. This is a test."

# Get probabilities for each character
probs = sat.predict_proba(text)
# Returns: numpy array of shape (len(text),) with boundary probabilities

# Find high-probability boundary positions
threshold = 0.5
boundaries = np.where(probs > threshold)[0]
print(f"Likely boundaries at positions: {boundaries}")

# Batch prediction (much faster for multiple texts)
texts = ["First text here.", "Second text.", "Third one."]
all_probs = list(sat.predict_proba(texts))
# Returns iterator of probability arrays
```

--------------------------------

### Train WtP model using configuration file

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Launches the WtP model training process using a specified configuration file. Training was performed on a TPUv3-8.

```bash
python wtpsplit/train/train.py configs/<config_name>.json
```

--------------------------------

### Initialize and Use WtP Model

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Demonstrates initializing a WtP model, optionally moving it to GPU, and splitting text into sentences. Supports batch processing for better performance.

```python
from wtpsplit import WtP

wtp = WtP("wtp-bert-mini")
# optionally run on GPU for better performance
# also supports TPUs via e.g. wtp.to("xla:0"), in that case pass `pad_last_batch=True` to wtp.split
wtp.half().to("cuda")

# returns ["Hello ", "This is a test."]
wtp.split("Hello This is a test.")

# returns an iterator yielding a lists of sentences for every text
# do this instead of calling wtp.split on every text individually for much better performance
wtp.split(["Hello This is a test.", "And some more texts..."])

# if you're using a model with language adapters, also pass a `lang_code`
wtp.split("Hello This is a test.", lang_code="en")

# depending on your usecase, adaptation to e.g. the Universal Dependencies style may give better results
# this always requires a language code
wtp.split("Hello This is a test.", lang_code="en", style="ud")
```

--------------------------------

### Reproduce Paper Training Runs

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Commands to launch training for base, sm, and LoRA models as used in the paper. Configuration files are located in the 'configs/' directory.

```bash
python3 wtpsplit/train/train.py configs/<config_name>.json
python3 wtpsplit/train/train_sm.py configs/<config_name>.json
python3 wtpsplit/train/train_lora.py configs/<config_name>.json
```

--------------------------------

### Initialize WtP Model with ONNX Support

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Shows how to initialize a WtP model with ONNX runtime providers for potentially faster inference, especially on GPU. Requires `onnxruntime` and `onnxruntime-gpu`.

```python
wtp = WtP("wtp-bert-mini", onnx_providers=["CUDAExecutionProvider"])
```

--------------------------------

### Initialize and use WtP models

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Load a WtP model and perform sentence splitting using the same interface as SaT models.

```python
from wtpsplit import WtP

wtp = WtP("wtp-bert-mini")
# similar functionality as for SaT models
wtp.split("This is a test This is another test.")
```

--------------------------------

### Create dummy dataset for WtP adaptation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Saves a dummy dataset in PyTorch format for WtP adaptation. This structure includes meta-information and data for training and testing sentences.

```python
import torch

torch.save(
    {
        "en": {
            "sentence": {
                "dummy-dataset": {
                    "meta": {
                        "train_data": ["train sentence 1", "train sentence 2"],
                    },
                    "data": [
                        "test sentence 1",
                        "test sentence 2",
                    ]
                }
            }
        }
    },
    "dummy-dataset.pth"
)
```

--------------------------------

### Initialize WtP Legacy Model

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Initialize the WtP legacy model for backward compatibility. A deprecation warning is shown by default, which can be suppressed by setting `ignore_legacy_warning=True`. GPU acceleration is available.

```python
from wtpsplit import WtP

# Initialize WtP model (shows deprecation warning by default)
wtp = WtP("wtp-bert-mini")

# Suppress legacy warning
wtp = WtP("wtp-bert-mini", ignore_legacy_warning=True)

# GPU acceleration
wtp.half().to("cuda")
```

--------------------------------

### Train LoRA Model

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Initiates the LoRA training process using a specified configuration file. Ensure the config points to the correct base model and training data.

```bash
python3 wtpsplit/train/train_lora.py configs/lora/lora_dummy_config.json
```

--------------------------------

### Basic Length Limiting with SaT

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Demonstrates how to limit segments to a maximum of 100 characters using the SaT model. Ensure the SaT model is initialized before use.

```python
from wtpsplit import SaT

sat = SaT("sat-3l-sm")

# Limit segments to 100 characters
segments = sat.split(text, max_length=100)
```

--------------------------------

### Paragraph Segmentation with WtP

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Demonstrates how to use WtP models to segment text into paragraphs, in addition to sentences. The paragraph segmentation threshold can be adjusted.

```python
# returns a list of paragraphs, each containing a list of sentences
# adjust the paragraph threshold via the `paragraph_threshold` argument.
wtp.split(text, do_paragraph_segmentation=True)
```

--------------------------------

### Selecting Segmentation Algorithm (Greedy vs. Viterbi)

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Compares the 'greedy' algorithm for faster, slightly suboptimal segmentation with the 'viterbi' algorithm for optimal results. The default is 'viterbi'.

```python
# Use greedy for speed (slightly suboptimal)
segments = sat.split(text, max_length=100, algorithm="greedy")

# Use viterbi for optimal results (default)
segments = sat.split(text, max_length=100, algorithm="viterbi")
```

--------------------------------

### Run WtP adaptation script

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Executes the adaptation script for WtP using a specified model path, evaluation data path, and including desired languages.

```bash
python3 wtpsplit/evaluation/adapt.py --model_path=benjamin/wtp-bert-mini --eval_data_path dummy-dataset.pth --include_langs=en
```

--------------------------------

### Adapt SaT Model with LoRA

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Initialize the SaT model with trained LoRA modules for adaptation to specific languages, domains, or styles. Optional GPU acceleration is also demonstrated.

```python
# use trained lora modules for strong adaptation to language & domain/style
sat_adapted = SaT("sat-3l", style_or_domain="ud", language="en")
sat_adapted.half().to("cuda") # optional, see above
sat_adapted.split("This is a test This is another test.")
# returns ['This is a test ', 'This is another test']
```

--------------------------------

### Prepare Training Data for LoRA

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Creates a dummy dataset in the required PyTorch format for LoRA training. Ensure sentences do not contain newline characters.

```python
import torch

torch.save(
    {
        "language_code": {
            "sentence": {
                "dummy-dataset": {
                    "meta": {
                        "train_data": ["train sentence 1", "train sentence 2"],
                    },
                    "data": [
                        "test sentence 1",
                        "test sentence 2",
                    ]
                }
            }
        }
    },
    "dummy-dataset.pth"
)
```

--------------------------------

### Initialize SaT models for specific performance needs

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Select between fast 3-layer models or high-performance 12-layer models. LoRA adaptation can be applied for domain-specific accuracy.

```python
from wtpsplit import SaT

# Speed-sensitive applications: 3-layer models
sat_fast = SaT("sat-3l")      # Base: 93.7% EN, 89.2% multilingual
sat_fast_sm = SaT("sat-3l-sm")  # SM variant: 96.5% EN, 93.5% multilingual

# Best performance: 12-layer models
sat_best = SaT("sat-12l")     # Base: 94.0% EN, 90.4% multilingual
sat_best_sm = SaT("sat-12l-sm") # SM variant: 97.4% EN, 96.0% multilingual (BEST)

# With LoRA adaptation (highest scores)
sat_lora = SaT("sat-12l", style_or_domain="ud", language="en")
# 97.3% EN, 95.9% multilingual

# Available models:
# sat-1l, sat-1l-sm    (smallest, fastest)
# sat-3l, sat-3l-sm    (recommended balance)
# sat-6l, sat-6l-sm    (medium)
# sat-9l               (larger)
# sat-12l, sat-12l-sm  (best quality)

# Supported languages (85 total):
# en, de, fr, es, it, pt, ru, zh, ja, ko, ar, hi, bn, ...
# Full list: af, am, ar, az, be, bg, bn, ca, ceb, cs, cy, da, de, el, en, eo,
#            es, et, eu, fa, fi, fr, fy, ga, gd, gl, gu, ha, he, hi, hu, hy,
#            id, ig, is, it, ja, jv, ka, kk, km, kn, ko, ku, ky, la, lt, lv,
#            mg, mk, ml, mn, mr, ms, mt, my, ne, nl, no, pa, pl, ps, pt, ro,
#            ru, si, sk, sl, sq, sr, sv, ta, te, tg, th, tr, uk, ur, uz, vi,
#            xh, yi, yo, zh, zu
```

--------------------------------

### Load Pre-trained Model

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Imports the necessary library and loads a pre-trained token classification model from Hugging Face. Replace 'segment-any-text/sat-3l-sm' with your desired model.

```python
import wtpsplit.models
from transformers import AutoModelForTokenClassification

model = AutoModelForTokenClassification.from_pretrained("segment-any-text/sat-3l-sm") # or some other model name; see https://huggingface.co/segment-any-text
```

--------------------------------

### Load WtP model with HuggingFace transformers

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Loads a WtP model using HuggingFace's `transformers` library. Ensure `wtpsplit.models` is imported to register custom models.

```python
# import wtpsplit.models to register the custom models 
# (character-level BERT w/ hash embeddings and canine with language adapters)
import wtpsplit.models
from transformers import AutoModelForTokenClassification

model = AutoModelForTokenClassification.from_pretrained("benjamin/wtp-bert-mini") # or some other model name
```

--------------------------------

### Enable ONNX Inference

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Configure the SaT model to use ONNX providers for improved inference speed on GPU.

```python
from wtpsplit import SaT

# Initialize with ONNX providers
sat = SaT("sat-3l-sm", ort_providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

# Split text (same API as PyTorch)
texts = ["This is a sentence. This is another sentence."] * 1000
sentences = list(sat.split(texts))

# Benchmark comparison (example results on RTX 3090):
# PyTorch GPU: ~144 ms for 1000 texts
# ONNX GPU: ~95 ms for 1000 texts (~50% faster)
```

--------------------------------

### Use SaT-SM Model for General Segmentation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Initialize and use the '-sm' variant of the SaT model for general sentence segmentation tasks. Optional GPU acceleration is also shown.

```python
# use our '-sm' models for general sentence segmentation tasks
sat_sm = SaT("sat-3l-sm")
sat_sm.half().to("cuda") # optional, see above
sat_sm.split("this is a test this is another test")
# returns ["this is a test ", "this is another test"]
```

--------------------------------

### Load custom WtP mixture and split text

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Loads a custom WtP mixture from a file and uses it to split text for a specific language and dataset style.

```python
from wtpsplit import WtP
import skops.io as sio

wtp = WtP(
    "wtp-bert-mini",
    mixtures=sio.load(
        "wtpsplit/.cache/wtp-bert-mini.skops",
        ["numpy.float32", "numpy.float64", "sklearn.linear_model._logistic.LogisticRegression"],
    ),
)

wtp.split("your text here", lang_code="en", style="dummy-dataset")
```

--------------------------------

### Initialize and Use SaT Model

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Initialize the SaT model and perform sentence segmentation on a single text string. For better performance, consider running on GPU or TPUs and batching multiple texts.

```python
from wtpsplit import SaT

sat = SaT("sat-3l")
# optionally run on GPU for better performance
# also supports TPUs via e.g. sat.to("xla:0"), in that case pass `pad_last_batch=True` to sat.split
sat.half().to("cuda")

sat.split("This is a test This is another test.")
# returns ["This is a test ", "This is another test."]
```

--------------------------------

### WtP Legacy Model with ONNX Support

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Utilize ONNX support for wtp-bert-* models for potentially faster inference. This is configured during the WtP model initialization by specifying the desired execution providers.

```python
from wtpsplit import WtP

# ONNX support for wtp-bert-* models
wtp = WtP("wtp-bert-mini", ort_providers=["CUDAExecutionProvider"])
```

--------------------------------

### Text Preservation with Constraints

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Demonstrates how to reconstruct the original text from segments when length constraints are applied. Ensure correct join method is used.

```python
original_text = "".join(segments)  # ← newlines may be inside segments
```

--------------------------------

### Prepare Custom LoRA Training Data

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Format training data for custom LoRA adapters. The data should be a dictionary structured by language, dataset name, and split (meta for training, data for testing), with sentences as list entries.

```python
import torch

# Step 1: Prepare training data format
# Each sentence is a separate list entry (no newlines within sentences)
data = {
    "en": {  # language code
        "sentence": {
            "my-dataset": {  # dataset name
                "meta": {
                    "train_data": [
                        "First training sentence.",
                        "Second training sentence.",
                        "Third one here."
                    ],
                },
                "data": [  # test data
                    "Test sentence one.",
                    "Test sentence two.",
                ]
            }
        }
    }
}

# Save training data
torch.save(data, "my-dataset.pth")
```

--------------------------------

### Compare PyTorch GPU vs ONNX GPU Performance

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Compares the performance of splitting a large list of texts using PyTorch on GPU versus ONNX Runtime on GPU. ONNX typically offers a speedup.

```python
>>> from wtpsplit import WtP
>>> texts = ["This is a sentence. This is another sentence."] * 1000

# PyTorch GPU
>>> model = WtP("wtp-bert-mini")
>>> model.half().to("cuda")
>>> %timeit list(model.split(texts))
272 ms ± 16.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# onnxruntime GPU
>>> model = WtP("wtp-bert-mini", ort_providers=["CUDAExecutionProvider"])
>>> %timeit list(model.split(texts))
198 ms ± 1.36 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

--------------------------------

### Adapt WtP with custom threshold for punctuation style

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Allows changing the threshold when using punctuation adaptation. Note that thresholds are inherently higher as they are not based on newline probability.

```python
wtp.split(text, lang_code="en", style="ud", threshold=0.7)
```

--------------------------------

### Load Wtpsplit models via HuggingFace

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Use AutoModelForTokenClassification to load models directly from the HuggingFace Hub. This approach allows access to model configurations and custom architectures.

```python
import wtpsplit.models
from transformers import AutoModelForTokenClassification, AutoConfig

# Load model from HuggingFace Hub
model = AutoModelForTokenClassification.from_pretrained("segment-any-text/sat-3l-sm")

# Access model configuration
config = AutoConfig.from_pretrained("segment-any-text/sat-3l-sm")
print(f"Model type: {config.model_type}")
print(f"Num labels: {config.num_labels}")

# For WtP models
model = AutoModelForTokenClassification.from_pretrained("benjamin/wtp-bert-mini")
```

--------------------------------

### Adapt WtP using threshold adaptation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Adapts the WtP model using a specified threshold, which can be obtained using `wtp.get_threshold`.

```python
threshold = wtp.get_threshold("en", "ud")

wtp.split(text, threshold=threshold)
```

--------------------------------

### Perform Sentence Segmentation with SaT

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Initialize the SaT model to perform sentence splitting on single or batch inputs, with support for GPU acceleration and threshold adjustment.

```python
from wtpsplit import SaT

# Initialize model (downloads from HuggingFace automatically)
sat = SaT("sat-3l-sm")

# Optional: Enable GPU acceleration
sat.half().to("cuda")

# Split a single text
sentences = sat.split("This is a test This is another test.")
# Returns: ["This is a test ", "This is another test."]

# Batch processing (much faster for multiple texts)
texts = ["First document. Second sentence.", "Another document here."]
results = list(sat.split(texts))
# Returns iterator of sentence lists: [["First document. ", "Second sentence."], ["Another document here."]]

# Use -sm models for general sentence segmentation (handles text without punctuation)
sat_sm = SaT("sat-3l-sm")
sentences = sat_sm.split("this is a test this is another test")
# Returns: ["this is a test ", "this is another test"]

# Adjust segmentation threshold (higher = more conservative)
sentences = sat.split("Hello world This is text", threshold=0.4)
```

--------------------------------

### WtP Legacy Model Splitting

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Perform basic text splitting using the WtP legacy model, which follows the same API as SaT. Supports language adapters and style adaptation (Universal Dependencies).

```python
from wtpsplit import WtP

wtp = WtP("wtp-bert-mini")

# Basic splitting (same API as SaT)
sentences = wtp.split("This is a test This is another test.")

# With language adapters
wtp.split("Hello This is a test.", lang_code="en")

# Style adaptation (Universal Dependencies)
wtp.split("Hello This is a test.", lang_code="en", style="ud")
```

--------------------------------

### Clone WtP repository

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Clones the WtP repository from GitHub to access its code and scripts.

```bash
git clone https://github.com/bminixhofer/wtpsplit
cd wtpsplit
```

--------------------------------

### Load Trained LoRA Adapter

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Load a trained LoRA adapter into the SaT model for domain-specific sentence segmentation. This allows the model to adapt to custom text characteristics with minimal data.

```python
from wtpsplit import SaT

sat_adapted = SaT("sat-3l", lora_path="/path/to/trained/adapter")
sentences = sat_adapted.split("Domain-specific text here")
```

--------------------------------

### POST /split

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Splits input text into segments based on specified length constraints and algorithm preferences.

```APIDOC
## POST /split

### Description
Splits a given text string into segments while adhering to minimum and maximum length constraints. The algorithm prioritizes natural sentence boundaries.

### Method
POST

### Endpoint
/split

### Parameters
#### Query Parameters
- **min_length** (int) - Optional - Minimum segment length (best effort).
- **max_length** (int) - Optional - Maximum segment length (strict).
- **algorithm** (str) - Optional - "viterbi" (optimal) or "greedy" (faster). Default: "viterbi".
- **prior_type** (str) - Optional - "uniform", "gaussian", "clipped_polynomial", or "lognormal". Default: "uniform".

#### Request Body
- **text** (str) - Required - The input text to be segmented.
- **prior_kwargs** (dict) - Optional - Parameters for the prior function, including 'target_length', 'spread', and 'lang_code'.

### Request Example
{
  "text": "The quick brown fox jumps. Pack my box with jugs.",
  "max_length": 100,
  "algorithm": "viterbi",
  "prior_type": "gaussian",
  "prior_kwargs": {
    "target_length": 50,
    "spread": 15
  }
}

### Response
#### Success Response (200)
- **segments** (array) - A list of segmented strings.

#### Response Example
{
  "segments": ["The quick brown fox jumps.", "Pack my box with jugs."]
}
```

--------------------------------

### Segmenting Text with Min and Max Lengths

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Shows how to enforce both minimum and maximum lengths for text segments, ensuring segments are between 20 and 100 characters. This provides more control over segment size.

```python
# Segments between 20-100 characters
segments = sat.split(text, min_length=20, max_length=100)
```

--------------------------------

### Compare PyTorch GPU vs. ONNX Runtime GPU Performance

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Benchmark the performance of the SaT model using PyTorch on GPU versus ONNX Runtime with CUDA execution providers. ONNX Runtime typically offers significant speed improvements.

```python
>>> from wtpsplit import SaT
>>> texts = ["This is a sentence. This is another sentence."] * 1000

# PyTorch GPU
>>> model_pytorch = SaT("sat-3l-sm")
>>> model_pytorch.half().to("cuda");
>>> %timeit list(model_pytorch.split(texts))
# 144 ms ± 252 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# quite fast already, but...

# onnxruntime GPU
>>> model_ort = SaT("sat-3l-sm", ort_providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
>>> %timeit list(model_ort.split(texts))
# 94.9 ms ± 165 μs per loop (mean ± std. dev. of 7 runs, 10 loops each
# ...this should be ~50% faster! (tested on RTX 3090)
```

--------------------------------

### Apply LoRA Adapters for Domain Adaptation

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Load pre-trained LoRA modules to adapt the model for specific domains, languages, or styles like code-switching.

```python
from wtpsplit import SaT

# Load with language-specific LoRA adapter
sat_lora = SaT("sat-3l", style_or_domain="ud", language="en")
sentences = sat_lora.split("Hello this is a test But this is different now")
# Returns: ['Hello this is a test ', 'But this is different now']

# Code-switching domains (multilingual content)
sat_cs = SaT("sat-12l", style_or_domain="code-switching", language="es-en")
sentences = sat_cs.split("in the morning over there cada vez que yo decía algo")

# Adapt threshold for LoRA models (they use higher thresholds)
sentences = sat_lora.split("Some text here", threshold=0.7)
```

--------------------------------

### Using Gaussian Prior for Preferred Segment Length

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Utilizes a Gaussian prior to encourage segments around a target length of 50 characters with a spread of 15. This helps in achieving more uniform segment sizes.

```python
# Prefer ~50 character segments
segments = sat.split(
    text,
    max_length=100,
    prior_type="gaussian",
    prior_kwargs={"target_length": 50, "spread": 15}
)
```

--------------------------------

### Inference with LoRA Adapted Model

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Loads a LoRA-adapted model for inference. Use the same model variant for inference as was used for training.

```python
sat_lora_adapted = SaT("model-used", lora_path="dummy_lora_path")
sat_lora_adapted.split("Some domains-specific or styled text")
```

--------------------------------

### Adapt WtP to UD, OPUS100, or Ersatz style using punctuation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Use punctuation adaptation for language-specific segmentation styles like Universal Dependencies. Requires a `lang_code`.

```python
wtp.split(text, lang_code="en", style="ud")
```

--------------------------------

### Adapt Segmentation Threshold with LoRA

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Adapt the segmentation threshold when using LoRA modules. Thresholds are generally higher for LoRA models.

```python
# works similarly for lora; but thresholds are higher
sat_lora.split("Hello this is a test But this is different now Now the next one starts looool", threshold=0.7)
```

--------------------------------

### Use Greedy Algorithm for Faster Segmentation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Utilize the greedy algorithm for minimally faster, though potentially less optimal, segmentation results.

```python
sat.split(text, max_length=120, algorithm="greedy")
```

--------------------------------

### Perform paragraph segmentation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Enable paragraph segmentation by setting the do_paragraph_segmentation flag to True.

```python
# returns a list of paragraphs, each containing a list of sentences
# adjust the paragraph threshold via the `paragraph_threshold` argument.
sat.split(text, do_paragraph_segmentation=True)
```

--------------------------------

### Predict sentence boundary probabilities for a specific style

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Calculates sentence boundary probabilities for a given text, language, and segmentation style.

```python
# returns sentence boundary probabilities for the given style
wtp.predict_proba(text, lang_code="en", style="ud")
```

--------------------------------

### Basic Text Segmentation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Use the split method for basic text segmentation. Ensure text is perfectly preserved by joining segments.

```python
assert "".join(segments) == text  # text is perfectly preserved
```

--------------------------------

### Language-Aware Defaults for Gaussian Prior (Chinese)

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Leverage language-specific defaults for target_length and spread by passing a lang_code. Chinese defaults to target_length=45, spread=15.

```python
# Chinese has shorter sentences → auto-uses target_length=45, spread=15
sat.split(text, max_length=100, prior_type="gaussian", 
          prior_kwargs={"lang_code": "zh"})
```

--------------------------------

### Language-Aware Defaults for Gaussian Prior (German)

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Leverage language-specific defaults for target_length and spread by passing a lang_code. German defaults to target_length=90, spread=35.

```python
# German has longer average sentences → auto-uses target_length=90, spread=35
sat.split(text, max_length=150, prior_type="gaussian", 
          prior_kwargs={"lang_code": "de"})
```

--------------------------------

### Batch Text Segmentation with SaT

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

For significantly improved performance, batch multiple text strings for segmentation instead of processing them individually.

```python
# do this instead of calling sat.split on every text individually for much better performance
sat.split(["This is a test This is another test.", "And some more texts..."])
# returns an iterator yielding lists of sentences for every text
```

--------------------------------

### Text Joining Without Input Newlines

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Demonstrates joining segments without newlines, typically used when WtP is enabled or split_on_input_newlines is set to False.

```python
original_text = "".join(segments)
```

--------------------------------

### Text Reconstruction Without Constraints

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Reconstruct original text from segments. Without constraints and with split_on_input_newlines=True, segments are joined by newlines.

```python
# Without constraints (SaT default with split_on_input_newlines=True):
original_text = "\n".join(segments)
```

--------------------------------

### Default Text Joining Behavior

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Shows the default behavior of joining segments with newlines when using SaT with split_on_input_newlines=True.

```python
original_text = "\n".join(segments)
```

--------------------------------

### Gaussian Prior for Segment Length Preference

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Use the Gaussian prior to prefer segments around a target length. Requires specifying target_length and spread.

```python
sat.split(text, max_length=100, prior_type="gaussian", 
          prior_kwargs={"target_length": 50, "spread": 10})
```

--------------------------------

### Segment Text with Gaussian Prior

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Use a Gaussian prior to influence segment length, preferring segments around a target length. Requires specifying the prior type and relevant keyword arguments.

```python
from wtpsplit import SaT

sat = SaT("sat-3l-sm")
text = "Long text with multiple sentences that need segmentation..."

# Gaussian prior: prefer segments around target_length
segments = sat.split(
    text,
    max_length=100,
    prior_type="gaussian",
    prior_kwargs={"target_length": 50, "spread": 15}
)
```

--------------------------------

### Adapt Segmentation Threshold

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Freely adapt the segmentation threshold. A higher threshold leads to more conservative segmentation.

```python
sat.split("This is a test This is another test.", threshold=0.4)
```

--------------------------------

### Text Reconstruction with Constraints

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Reconstruct original text from segments. When using length constraints, segments may contain newlines and require post-processing.

```python
# With constraints (max_length or min_length):
original_text = "".join(segments)  # segments may contain newlines
```

--------------------------------

### Segment Text with Language-Aware Gaussian Prior

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Leverage language-aware defaults for the Gaussian prior, automatically adjusting target length and spread based on corpus statistics for the specified language code. This simplifies configuration for different languages.

```python
from wtpsplit import SaT

sat = SaT("sat-3l-sm")
text = "Long text with multiple sentences that need segmentation..."

# Language-aware defaults (uses corpus statistics)
# German has longer sentences → auto-uses target_length=90, spread=35
segments = sat.split(
    text,
    max_length=150,
    prior_type="gaussian",
    prior_kwargs={"lang_code": "de"}
)

# Chinese has shorter sentences → auto-uses target_length=45, spread=15
segments = sat.split(
    text,
    max_length=100,
    prior_type="gaussian",
    prior_kwargs={"lang_code": "zh"}
)
```

--------------------------------

### Perform Length-Constrained Segmentation

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Use Viterbi dynamic programming or greedy algorithms to enforce minimum and maximum segment length constraints.

```python
from wtpsplit import SaT

sat = SaT("sat-3l-sm")

text = (
    "In the beginning God created the heaven and the earth. "
    "And the earth was without form, and void; and darkness was upon the face of the deep. "
    "And the Spirit of God moved upon the face of the waters. "
    "And God said, Let there be light: and there was light."
)

# Basic max_length constraint (strict enforcement)
segments = sat.split(text, max_length=120)
# All segments guaranteed <= 120 characters
for s in segments:
    print(f"[{len(s):3d} chars] {s}")

# Both min and max length
segments = sat.split(text, min_length=80, max_length=200)

# Greedy algorithm (faster but less optimal)
segments = sat.split(text, max_length=120, algorithm="greedy")
```

--------------------------------

### Automatic Language Defaults with LoRA

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

When using LoRA with a specified language, language-specific defaults for segmentation parameters are automatically applied.

```python
sat = SaT("sat-3l", style_or_domain="ud", language="de")
sat.split(text, max_length=150, prior_type="gaussian")  # auto-uses German defaults
```

--------------------------------

### Clipped Polynomial Prior for Strict Length Adherence

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Use the clipped_polynomial prior for segments that must be very close to the target length, with a hard cutoff at +/- spread.

```python
sat.split(text, max_length=100, prior_type="clipped_polynomial", 
          prior_kwargs={"target_length": 60, "spread": 25})
```

--------------------------------

### Segment Text with Log-normal Prior

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Utilize a log-normal prior for right-skewed segment length distribution, allowing for more tolerance of longer segments. Specify the prior type and its keyword arguments.

```python
from wtpsplit import SaT

sat = SaT("sat-3l-sm")
text = "Long text with multiple sentences that need segmentation..."

# Log-normal prior: right-skewed (more tolerant of longer segments)
segments = sat.split(
    text,
    max_length=100,
    prior_type="lognormal",
    prior_kwargs={"target_length": 70, "spread": 25}
)
```

--------------------------------

### Enforce Min and Max Length Segmentation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Enforce both minimum and maximum segment lengths during text segmentation.

```python
sat.split(text, min_length=80, max_length=200)
```

--------------------------------

### Predict newline probabilities

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README_WTP.md

Calculates the newline probabilities for a given text. Supports batching.

```python
# returns newline probabilities (supports batching!)
wtp.predict_proba(text)
```

--------------------------------

### Apply length-constrained segmentation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Use the max_length parameter to restrict the character length of output segments.

```python
from wtpsplit import SaT

sat = SaT("sat-3l-sm")

text = (
    "In the beginning God created the heaven and the earth. "
    "And the earth was without form, and void; and darkness was upon the face of the deep. "
    "And the Spirit of God moved upon the face of the waters. "
    "And God said, Let there be light: and there was light. "
    "And God saw the light, that it was good: and God divided the light from the darkness. "
    "And God called the light Day, and the darkness he called Night. "
    "And the evening and the morning were the first day."
)

# Split with a maximum segment length of 120 characters
segments = sat.split(text, max_length=120)
for i, s in enumerate(segments):
    print(f"[{len(s):3d} chars] {s}")
# [ 55 chars] In the beginning God created the heaven and the earth. 
# [ 86 chars] And the earth was without form, and void; and darkness was upon the face of the deep. 
# [112 chars] And the Spirit of God moved upon the face of the waters. And God said, Let there be light: and there was light. 
```

--------------------------------

### Log-normal Prior for Right-Skewed Length Preference

Source: https://github.com/segment-any-text/wtpsplit/blob/main/README.md

Employ the log-normal prior for a right-skewed preference, making the model more tolerant of longer segments. Requires target_length and spread.

```python
sat.split(text, max_length=100, prior_type="lognormal", 
          prior_kwargs={"target_length": 70, "spread": 25})
```

--------------------------------

### Uniform Prior Function

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Defines a uniform prior where all lengths up to max_length are equally preferred, with a hard cutoff. Use for simple length limiting.

```python
Prior(length) = 1.0 if length ≤ max_length else 0.0
```

--------------------------------

### Paragraph Segmentation

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Segment text into paragraphs, where each paragraph is a list of sentences. This functionality is enabled by setting `do_paragraph_segmentation=True`. The `paragraph_threshold` parameter can be used to adjust sensitivity.

```python
from wtpsplit import SaT

sat = SaT("sat-3l-sm")

text = """First paragraph here. Multiple sentences.
Second paragraph starts. Another sentence here.
Third paragraph with content."""

# Returns nested list: [[sentences in para1], [sentences in para2], ...]
paragraphs = sat.split(text, do_paragraph_segmentation=True)

for i, para in enumerate(paragraphs):
    print(f"Paragraph {i+1}:")
    for sentence in para:
        print(f"  - {sentence}")

# Adjust paragraph detection sensitivity
paragraphs = sat.split(text, do_paragraph_segmentation=True, paragraph_threshold=0.7)
```

--------------------------------

### Clipped Polynomial Prior Function

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Defines a clipped polynomial prior that peaks at target_length and clips to zero at ±spread. Use for strong enforcement of segment length windows.

```python
Prior(length) = max(1 - (1/spread²) × (length - target_length)², 0)
```

--------------------------------

### Segment Text with Clipped Polynomial Prior

Source: https://context7.com/segment-any-text/wtpsplit/llms.txt

Employ a clipped polynomial prior, which enforces a hard cutoff for segment lengths at a specified spread from the target length. This requires setting the prior type and its associated keyword arguments.

```python
from wtpsplit import SaT

sat = SaT("sat-3l-sm")
text = "Long text with multiple sentences that need segmentation..."

# Clipped polynomial: hard cutoff at ±spread from target
segments = sat.split(
    text,
    max_length=100,
    prior_type="clipped_polynomial",
    prior_kwargs={"target_length": 60, "spread": 25}
)
```

--------------------------------

### Gaussian Prior Function

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Defines a Gaussian prior peaking at target_length and falling off based on spread. Recommended for preferring specific chunk sizes.

```python
Prior(length) = exp(-0.5 × ((length - target_length) / spread)²)
```

--------------------------------

### Viterbi Algorithm DP Calculation

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

The dynamic programming recurrence for the Viterbi algorithm to find the globally optimal segmentation. It calculates the best score for text up to index i by considering previous split points j.

```python
dp[i] = max over j of: dp[j] + log(Prior(i-j)) + log(P[i-1])

Where j ranges from max(0, i-max_length) to i-min_length
```

--------------------------------

### Log-Normal Prior Function

Source: https://github.com/segment-any-text/wtpsplit/blob/main/docs/LENGTH_CONSTRAINTS.md

Defines a log-normal prior, resulting in a right-skewed distribution. More tolerant of longer segments than shorter ones. Use when leniency with longer segments is desired.

```python
Prior(length) = exp(-0.5 × ((log(length) - μ) / σ)²) / length
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.