### Development Environment Setup

Source: https://github.com/natasha/razdel/blob/master/README.md

Commands to set up a Python virtual environment and install the library in development mode.

```bash
python -m venv ~/
.venvs/natasha-razdel
source ~/.venvs/natasha-razdel/bin/activate

pip install -r requirements/dev.txt
pip install -e .
```

--------------------------------

### Install Razdel via pip

Source: https://github.com/natasha/razdel/blob/master/README.md

Install the library using the standard pip package manager.

```bash
$ pip install razdel
```

--------------------------------

### Using Substring Objects from Tokenize and Sentenize

Source: https://context7.com/natasha/razdel/llms.txt

Demonstrates the attributes (start, stop, text) of Substring objects returned by tokenize and sentenize. Shows how to verify positions, unpack attributes, compare objects, and reconstruct text spans.

```python
from razdel import tokenize, sentenize

# Substring objects from tokenize
text = 'Привет мир'
tokens = list(tokenize(text))

token = tokens[0]
print(f"Text: {token.text}")      # Text: Привет
print(f"Start: {token.start}")    # Start: 0
print(f"Stop: {token.stop}")      # Stop: 6

# Verify positions match original text
assert text[token.start:token.stop] == token.text

# Substring objects are iterable (yields start, stop, text)
start, stop, text_value = token
print(f"Unpacked: {start}, {stop}, {text_value}")
# Unpacked: 0, 6, Привет

# Substring objects are comparable
token1, token2 = tokens
print(token1 == token2)  # False

# Reconstruct text spans using positions
original = 'Москва — столица России.'
sentences = list(sentenize(original))
for sent in sentences:
    # Positions allow exact slice reconstruction
    reconstructed = original[sent.start:sent.stop]
    assert reconstructed == sent.text
```

--------------------------------

### Batch Processing with Global Position Tracking

Source: https://context7.com/natasha/razdel/llms.txt

A function to process a document, extracting tokens and adjusting their start and stop positions to reflect their global offsets within the entire document. It also tracks the sentence's starting position.

```python
from razdel import tokenize, sentenize

# Batch processing with position tracking
def process_document(text):
    """Extract tokens with global positions."""
    results = []
    for sentence in sentenize(text):
        for token in tokenize(sentence.text):
            # Adjust token positions to global document positions
            global_start = sentence.start + token.start
            global_stop = sentence.start + token.stop
            results.append({
                'text': token.text,
                'start': global_start,
                'stop': global_stop,
                'sentence_start': sentence.start
            })
    return results

# Example usage
doc = "Привет! Как дела? Всё хорошо."
annotations = process_document(doc)
for ann in annotations[:5]:
    print(ann)
# {'text': 'Привет', 'start': 0, 'stop': 6, 'sentence_start': 0}
# {'text': '!', 'start': 6, 'stop': 7, 'sentence_start': 0}
# {'text': 'Как', 'start': 8, 'stop': 11, 'sentence_start': 8}
# ...
```

--------------------------------

### Running Tests

Source: https://github.com/natasha/razdel/blob/master/README.md

Commands to execute the test suite for the library, including integration tests.

```bash
make test
make int  # 2000 integration tests
```

--------------------------------

### Run non-trivial token tests

Source: https://github.com/natasha/razdel/blob/master/README.md

Generate recall and precision metrics for different tokenization methods and save to a file.

```bash
pv data/*_tokens.txt | razdel-ctl gen --recall | razdel-ctl diff space_tokenize > tests.txt
pv data/*_tokens.txt | razdel-ctl gen --precision | razdel-ctl diff re_tokenize >> tests.txt
```

--------------------------------

### Debug mystem errors

Source: https://github.com/natasha/razdel/blob/master/README.md

Analyze tokenization differences against moses_tokenize using a sample of data.

```bash
cat syntag_tokens.txt | razdel-ctl sample 1000 | razdel-ctl gen | razdel-ctl diff --show moses_tokenize | less
```

--------------------------------

### Benchmark razdel performance

Source: https://github.com/natasha/razdel/blob/master/README.md

Measure tokenization throughput using a sample of 10,000 tokens.

```bash
cat data/*_tokens.txt | razdel-ctl sample 10000 | pv -l | razdel-ctl gen | razdel-ctl diff tokenize | wc -l
```

--------------------------------

### Update project version

Source: https://github.com/natasha/razdel/blob/master/README.md

Commit and tag the current project state for version release.

```bash
git commit -am 'Up version'
git tag v0.5.0

git push
git push --tags
```

--------------------------------

### Compare razdel and moses tokenization

Source: https://github.com/natasha/razdel/blob/master/README.md

Perform a diff between razdel and moses tokenization outputs on a sample of data.

```bash
cat data/*_tokens.txt | razdel-ctl sample 1000 | razdel-ctl gen | razdel-ctl up tokenize | razdel-ctl diff moses_tokenize | less
```

--------------------------------

### Update integration tests

Source: https://github.com/natasha/razdel/blob/master/README.md

Update the sentence tokenization test data by processing sents.txt.

```bash
cd tests/data/
pv sents.txt | razdel-ctl up sentenize > t; mv t sents.txt
```

--------------------------------

### Sentence segmentation with Razdel

Source: https://github.com/natasha/razdel/blob/master/README.md

Use the sentenize function to split a block of text into individual sentences.

```python
>>> from razdel import sentenize

>>> text = '''
... - "Так в чем же дело?" - "Не ра-ду-ют".
... И т. д. и т. п. В общем, вся газета
... '''

>>> list(sentenize(text))
[Substring(1, 23, '- "Так в чем же дело?"'),
 Substring(24, 40, '- "Не ра-ду-ют".'),
 Substring(41, 56, 'И т. д. и т. п.'),
 Substring(57, 76, 'В общем, вся газета')]
```

--------------------------------

### Debugging Sentence Segmentation Rules

Source: https://context7.com/natasha/razdel/llms.txt

Enables debug mode for the sentenize function to visualize which rules are applied at each potential sentence split point. Aids in understanding and debugging sentence segmentation.

```python
from razdel import tokenize, sentenize

# Debug sentence segmentation
text2 = 'И т. д. Продолжение.'
debug_sentenizer = sentenize.debug
result2 = list(debug_sentenizer(text2))
# Output shows which rules fire for each potential split point
# 'И т' | '.' | ' д'
#      join sokr_left
# 'т. д' | '.' | ' Прод'
#      join sokr_left
```

--------------------------------

### Debugging Tokenization Rules

Source: https://context7.com/natasha/razdel/llms.txt

Enables debug mode for the tokenize function to inspect internal rule decisions made during the segmentation process. Useful for troubleshooting tokenization behavior.

```python
from razdel import tokenize, sentenize

# Debug tokenization - shows rule decisions
text = 'что-то'
debug_tokenizer = tokenize.debug
result = list(debug_tokenizer(text))
# Output shows split decisions:
# 'что' | '-' | 'то'
#      join dash
```

--------------------------------

### Tokenize Russian text with Razdel

Source: https://github.com/natasha/razdel/blob/master/README.md

Use the tokenize function to split a string into individual tokens, returning Substring objects with position indices.

```python
>>> from razdel import tokenize

>>> tokens = list(tokenize('Кружка-термос на 0.5л (50/64 см³, 516;...)'))
>>> tokens
[Substring(0, 13, 'Кружка-термос'),
 Substring(14, 16, 'на'),
 Substring(17, 20, '0.5'),
 Substring(20, 21, 'л'),
 Substring(22, 23, '(')
 ...]
 
>>> [_.text for _ in tokens]
['Кружка-термос', 'на', '0.5', 'л', '(', '50/64', 'см³', ',', '516', ';', '...', ')']
```

--------------------------------

### Memory-Efficient Processing of Large Texts

Source: https://context7.com/natasha/razdel/llms.txt

Illustrates how to process large text documents efficiently by iterating over generators directly, avoiding the memory overhead of converting them to lists. This approach is suitable for large corpora.

```python
from razdel import tokenize, sentenize

# Process large document efficiently
large_text = """
Москва — столица России и крупнейший город страны.
Население составляет более 12 миллионов человек.
Город был основан в 1147 году.
""" * 1000  # Simulate large document

# Memory-efficient sentence processing
sentence_count = 0
for sentence in sentenize(large_text):
    sentence_count += 1
    # Process each sentence without loading all into memory
    tokens = list(tokenize(sentence.text))
    # Do something with tokens...

print(f"Processed {sentence_count} sentences")
```

--------------------------------

### Tokenize Russian Text

Source: https://context7.com/natasha/razdel/llms.txt

Use the tokenize function to split Russian text into words, numbers, and punctuation. Handles hyphenated words, numbers, and emoticons.

```python
from razdel import tokenize

# Basic tokenization
text = 'Кружка-термос на 0.5л (50/64 см³, 516;...)'
tokens = list(tokenize(text))

# Each token is a Substring(start, stop, text) object
print(tokens)
# [Substring(0, 13, 'Кружка-термос'),
#  Substring(14, 16, 'на'),
#  Substring(17, 20, '0.5'),
#  Substring(20, 21, 'л'),
#  Substring(22, 23, '('),
#  Substring(23, 28, '50/64'),
#  Substring(29, 32, 'см³'),
#  Substring(32, 33, ','),
#  Substring(34, 37, '516'),
#  Substring(37, 38, ';'),
#  Substring(38, 41, '...'),
#  Substring(41, 42, ')')]
```

```python
# Extract just the text from tokens
token_texts = [_.text for _ in tokens]
print(token_texts)
# ['Кружка-термос', 'на', '0.5', 'л', '(', '50/64', 'см³', ',', '516', ';', '...', ')']
```

```python
# Access token positions for annotation
for token in tokens:
    print(f"'{token.text}' at positions {token.start}:{token.stop}")
# 'Кружка-термос' at positions 0:13
# 'на' at positions 14:16
# '0.5' at positions 17:20
# ...
```

```python
# Handles underscored compound words
text2 = 'К_тому_же это важно'
tokens2 = [_.text for _ in tokenize(text2)]
print(tokens2)
# ['К_тому_же', 'это', 'важно']
```

```python
# Preserves emoticons as single tokens
text3 = 'Привет :))) Как дела?'
tokens3 = [_.text for _ in tokenize(text3)]
print(tokens3)
# ['Привет', ':)))', 'Как', 'дела', '?']
```

--------------------------------

### Tokenize Function

Source: https://context7.com/natasha/razdel/llms.txt

The `tokenize` function splits Russian text into individual tokens including words, numbers, and punctuation marks. It returns a generator of `Substring` objects containing the token text and its character positions in the original string.

```APIDOC
## tokenize

### Description
Splits Russian text into individual tokens (words, numbers, punctuation). Handles hyphenated words, numbers, fractions, underscores, and punctuation.

### Method
Function Call

### Parameters
None

### Request Example
```python
from razdel import tokenize

text = 'Кружка-термос на 0.5л (50/64 см³, 516;...)'
tokens = list(tokenize(text))
print(tokens)
```

### Response
#### Success Response (Generator of Substring objects)
- **start** (int) - The starting character index of the token.
- **stop** (int) - The ending character index of the token.
- **text** (str) - The extracted token text.

#### Response Example
```json
[
  Substring(0, 13, 'Кружка-термос'),
  Substring(14, 16, 'на'),
  Substring(17, 20, '0.5'),
  Substring(20, 21, 'л'),
  Substring(22, 23, '('),
  Substring(23, 28, '50/64'),
  Substring(29, 32, 'см³'),
  Substring(32, 33, ','),
  Substring(34, 37, '516'),
  Substring(37, 38, ';'),
  Substring(38, 41, '...'),
  Substring(41, 42, ')')
]
```
```

--------------------------------

### Sentenize Function

Source: https://context7.com/natasha/razdel/llms.txt

The `sentenize` function splits Russian text into sentences, handling complex cases like abbreviations, initials, dialog markers, quoted speech, and numbered lists. It returns a generator of `Substring` objects with sentence text and character positions.

```APIDOC
## sentenize

### Description
Splits Russian text into sentences. Handles abbreviations, initials, dialog markers, quoted speech, and numbered lists.

### Method
Function Call

### Parameters
None

### Request Example
```python
from razdel import sentenize

text = '''
- "Так в чем же дело?" - "Не ра-ду-ют".
И т. д. и т. п. В общем, вся газета
'''
sentences = list(sentenize(text))
print(sentences)
```

### Response
#### Success Response (Generator of Substring objects)
- **start** (int) - The starting character index of the sentence.
- **stop** (int) - The ending character index of the sentence.
- **text** (str) - The extracted sentence text.

#### Response Example
```json
[
  Substring(1, 23, '- "Так в чем же дело?"'),
  Substring(24, 40, '- "Не ра-ду-ют".'),
  Substring(41, 56, 'И т. д. и т. п.'),
  Substring(57, 76, 'В общем, вся газета')
]
```
```

--------------------------------

### Sentenize Russian Text

Source: https://context7.com/natasha/razdel/llms.txt

Use the sentenize function to split Russian text into sentences. Handles abbreviations, initials, dialog markers, quoted speech, and numbered lists.

```python
from razdel import sentenize

# Basic sentence segmentation
text = '''
- "Так в чем же дело?" - "Не ра-ду-ют".
И т. д. и т. п. В общем, вся газета
'''

sentences = list(sentenize(text))
print(sentences)
# [Substring(1, 23, '- "Так в чем же дело?"'),
#  Substring(24, 40, '- "Не ра-ду-ют".'),
#  Substring(41, 56, 'И т. д. и т. п.'),
#  Substring(57, 76, 'В общем, вся газета')]
```

```python
# Extract just the sentence text
sentence_texts = [_.text for _ in sentences]
print(sentence_texts)
# ['- "Так в чем же дело?"',
#  '- "Не ра-ду-ют".',
#  'И т. д. и т. п.',
#  'В общем, вся газета']
```

```python
# Handles abbreviations correctly (doesn't split on abbreviation periods)
text2 = 'Директор фирмы Чарльз Дж. Филлипс прибыл вчера. Встреча состоится завтра.'
sentences2 = [_.text for _ in sentenize(text2)]
print(sentences2)
# ['Директор фирмы Чарльз Дж. Филлипс прибыл вчера.',
#  'Встреча состоится завтра.']
```

```python
# Handles numbered lists
text3 = '1. Первый пункт важен. 2. Второй пункт тоже. 3. И третий.'
sentences3 = [_.text for _ in sentenize(text3)]
print(sentences3)
# ['1. Первый пункт важен.',
#  '2. Второй пункт тоже.',
#  '3. И третий.']
```

```python
# Handles dialog with dashes
text4 = '— Ты ей скажи, что я ей гостинца дам. — А мне дашь?'
sentences4 = [_.text for _ in sentenize(text4)]
print(sentences4)
# ['— Ты ей скажи, что я ей гостинца дам.',
#  '— А мне дашь?']
```

```python
# Handles quoted speech and complex punctuation
text5 = 'Он сказал: "Это невозможно!" Но она не поверила.'
sentences5 = [_.text for _ in sentenize(text5)]
print(sentences5)
# ['Он сказал: "Это невозможно!"',
#  'Но она не поверила.']
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.