```
```bash
echo 'WeihnachtenFrüher war mehr Lametta!
' | somajo-tokenizer --xml -
```
--------------------------------
### Tokenize Text File with Empty Lines
Source: https://github.com/tsproisl/somajo/blob/master/doc/source/somajo.md
Demonstrates tokenizing a text file where paragraphs are separated by empty lines. It prints each token with its class and extra information, followed by an empty line for each sentence.
```APIDOC
## Tokenize Text File with Empty Lines
### Description
Tokenizes a text file with paragraphs separated by empty lines. Outputs each token on a new line with its class and extra information, and an empty line after each sentence.
### Method
```python
tokenizer.tokenize_text_file()
```
### Endpoint
N/A (Library function)
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
```python
with open("example_empty_lines.txt") as f:
print(f.read())
sentences = tokenizer.tokenize_text_file("example_empty_lines.txt", paragraph_separator="single_newlines")
for sentence in sentences:
for token in sentence:
print(f"{token.text} {token.token_class} {token.extra_info}")
print()
```
### Response
#### Success Response (200)
List of sentences, where each sentence is a list of `Token` objects.
#### Response Example
```
Heyi regular SpaceAfter=No
:) emoticon
Was regular
machst regular
du regular
morgen regular
Abend regular SpaceAfter=No
?! symbol
Lust regular
auf regular
Film regular SpaceAfter=No
? symbol SpaceAfter=No
;-) emoticon
```
```
--------------------------------
### tokenize_text() - With Token Metadata
Source: https://context7.com/tsproisl/somajo/llms.txt
Demonstrates how to access token metadata such as token class and spacing information.
```APIDOC
## tokenize_text() - With Token Metadata
### Description
Token objects contain additional metadata including token class, spacing information, and original spelling.
### Method
`tokenize_text(paragraphs)`
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
- **paragraphs** (iterable of strings) - Required - A list or other iterable of paragraph strings.
### Request Example
```python
from somajo import SoMaJo
tokenizer = SoMaJo("de_CMC", split_camel_case=True)
paragraphs = ["Heyi:) Was machst du?! Lust auf Film?;-)"]
sentences = tokenizer.tokenize_text(paragraphs)
for sentence in sentences:
for token in sentence:
print(f"{token.text:15} class={token.token_class:12} {token.extra_info}")
print()
```
### Response
#### Success Response (200)
A generator yielding lists of `Token` objects, where each `Token` object has attributes like `text`, `token_class`, and `extra_info`.
#### Response Example
```
Heyi class=regular SpaceAfter=No
:) class=emoticon
Was class=regular
machst class=regular
du class=regular SpaceAfter=No
?! class=symbol
Lust class=regular
auf class=regular
Film class=regular SpaceAfter=No
? class=symbol SpaceAfter=No
;-) class=emoticon
```
```
--------------------------------
### Tokenize XML Data (Default Settings)
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Tokenizes XML data, splitting sentences based on provided end-of-sentence tags. Prints each token on a new line, with an empty line separating sentences.
```python
>>> xml = "Heyi:)
Was machst du morgen Abend?! Lust auf Film?;-)
"
>>> eos_tags = "title h1 h2 h3 h4 h5 h6 p br hr div ol ul dl table".split()
>>> tokenizer = SoMaJo("de_CMC")
>>> sentences = tokenizer.tokenize_xml(xml, eos_tags)
>>> for sentence in sentences:
... for token in sentence:
... print(token.text)
... print()
...
Heyi
:)
Was
machst
du
morgen
Abend
?!
Lust
auf
Film
?
;-)
```
--------------------------------
### Initialize SoMaJo Tokenizer for German
Source: https://github.com/tsproisl/somajo/blob/master/doc/source/somajo.md
Initializes the SoMaJo tokenizer with German language rules. This is useful for processing German text with default sentence splitting enabled.
```python
tokenizer = SoMaJo("de_CMC")
```
--------------------------------
### Parallel Text File Tokenization with SoMaJo
Source: https://context7.com/tsproisl/somajo/llms.txt
Use the 'parallel' parameter to leverage multiple CPU cores for faster tokenization of large text files. This can also be used with tokenize_text(), tokenize_xml(), and tokenize_xml_file().
```python
from somajo import SoMaJo
tokenizer = SoMaJo("de_CMC")
# Process large files with multiple workers
sentences = tokenizer.tokenize_text_file(
"large_file.txt",
paragraph_separator="empty_lines",
parallel=4 # Use 4 worker processes
)
for sentence in sentences:
print(" ".join([token.text for token in sentence]))
# Also works with tokenize_text(), tokenize_xml(), and tokenize_xml_file()
paragraphs = ["..." for _ in range(1000)] # Many paragraphs
sentences = tokenizer.tokenize_text(paragraphs, parallel=4)
```
--------------------------------
### Tokenize XML file with sentence splitting
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Tokenizes an XML file and splits it into sentences based on provided end-of-sentence tags. Prints each token on a new line, followed by an empty line after each sentence.
```python
>>> with open("example.xml") as f:
... print(f.read())
...
Heyi:)
Was machst du morgen Abend?! Lust auf Film?;-)
>>> eos_tags = "title h1 h2 h3 h4 h5 h6 p br hr div ol ul dl table".split()
>>> tokenizer = SoMaJo("de_CMC")
>>> sentences = tokenizer.tokenize_xml_file("example.xml", eos_tags)
>>> for sentence in sentences:
... for token in sentence:
... print(token)
... print()
...
Heyi
:)
Was
machst
du
morgen
Abend
?!
Lust
auf
Film
?
;-)
```
--------------------------------
### Initialize SoMaJo Tokenizer for German and English
Source: https://context7.com/tsproisl/somajo/llms.txt
Configure language-specific rules and tokenization options when creating a SoMaJo tokenizer instance. Supports German ('de_CMC') and English ('en_PTB') with options for camel case splitting, sentence splitting, XML sentence wrapping, and character offset computation.
```python
from somajo import SoMaJo
# German tokenizer with camel case splitting and sentence splitting
tokenizer = SoMaJo(
language="de_CMC", # 'de_CMC' for German, 'en_PTB' for English
split_camel_case=True, # Split camelCase words
split_sentences=True, # Enable sentence splitting (default)
xml_sentences=None, # Set to 's' to wrap sentences in tags
character_offsets=False # Compute character offsets for stand-off tokenization
)
# English tokenizer
english_tokenizer = SoMaJo(
language="en_PTB",
split_camel_case=False
)
```
--------------------------------
### Tokenize and Split Sentences
Source: https://github.com/tsproisl/somajo/blob/master/README.md
This command tokenizes input and splits the text into sentences. Use '-' to read from standard input.
```bash
somajo-tokenizer --split-sentences
```
```bash
echo 'Palim, Palim! Ich hätte gerne eine Flasche Pommes Frites.' | somajo-tokenizer --split-sentences -
```
--------------------------------
### Tokenize Text with Detailed Output
Source: https://github.com/tsproisl/somajo/blob/master/doc/source/somajo.md
Tokenizes text and prints each token's text, class, and extra information, with an empty line separating sentences. This provides a detailed view of the tokenization process, including token types and any associated metadata.
```python
sentences = tokenizer.tokenize_text(paragraphs)
for sentence in sentences:
for token in sentence:
print(f"{token.text} {token.token_class} {token.extra_info}")
print()
```
--------------------------------
### Tokenize XML Data - Strip Tags and Print Sentences
Source: https://github.com/tsproisl/somajo/blob/master/doc/source/somajo.md
Tokenizes XML data, strips XML tags from the output, and prints one sentence per line, joining tokens with spaces.
```python
>>> sentences = tokenizer.tokenize_xml(xml, eos_tags, strip_tags=True)
>>> for sentence in sentences:
... print(" ".join([token.text for token in sentence]))
...
Heyi :)
Was machst du morgen Abend ?!
Lust auf Film ? ;-)
```
--------------------------------
### CLI Wrapping Sentences in XML Tags
Source: https://context7.com/tsproisl/somajo/llms.txt
Wrap tokenized sentences in specified XML tags when processing text files via the command line.
```bash
# Wrap sentences in XML tags
somajo-tokenizer --sentence-tag s document.txt
```
--------------------------------
### Tokenize Text with XML Sentence Delimiters
Source: https://context7.com/tsproisl/somajo/llms.txt
Use the `xml_sentences` parameter in the SoMaJo constructor to wrap sentences in XML tags (e.g., '') instead of separating them with empty lines. This is useful for integrating tokenized output with XML-based workflows.
```python
from somajo import SoMaJo
tokenizer = SoMaJo("de_CMC", xml_sentences="s")
paragraphs = ["Heyi:) Wie gehts?"]
sentences = tokenizer.tokenize_text(paragraphs)
for sentence in sentences:
for token in sentence:
print(token.text)
```
--------------------------------
### CLI Outputting Character Offsets
Source: https://context7.com/tsproisl/somajo/llms.txt
Output character offsets for each token when using the command-line interface, useful for stand-off annotation.
```bash
# Output character offsets for stand-off annotation
echo 'Test text' | somajo-tokenizer --character-offsets -
```
--------------------------------
### Tokenize XML Data (Delimit Sentences with Tags)
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Tokenizes XML data and explicitly delimits sentences using specified XML tags (e.g., ''). Prints each token on a new line, with an empty line separating sentences.
```python
>>> tokenizer = SoMaJo("de_CMC", xml_sentences="s")
>>> sentences = tokenizer.tokenize_xml(xml, eos_tags)
>>> for sentence in sentences:
... for token in sentence:
... print(token.text)
... print()
...
Heyi
:)
Was
machst
du
morgen
Abend
?!
Lust
auf
Film
?
;-)
```
--------------------------------
### tokenize_text_file Method
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Information on tokenizing content directly from a text file.
```APIDOC
## POST /tokenize_text_file
### Description
Split the contents of a text file into sequences of tokens. This method is suitable for processing large text files efficiently.
### Method
POST
### Endpoint
/tokenize_text_file
### Parameters
#### Query Parameters
- **parallel** (int) - Optional - Default: 1 - Number of processes to use for parallel processing.
#### Request Body
- **text_file** (str or file-like object) - Required - Either a filename or a file-like object containing the text to be tokenized.
- **paragraph_separator** (str) - Required - Specifies how paragraphs are separated in the input file. Accepted values: 'single_newlines' (one paragraph per line) or 'empty_lines' (paragraphs separated by blank lines).
### Request Example
```json
{
"text_file": "path/to/your/textfile.txt",
"paragraph_separator": "empty_lines"
}
```
### Response
#### Success Response (200)
- **list** - Yields a list of `Token` objects for each sentence or paragraph.
```
--------------------------------
### Tokenize English Text (Penn Treebank)
Source: https://github.com/tsproisl/somajo/blob/master/README.md
Specify the English Penn Treebank tokenization guideline using the -l or --language option.
```bash
somajo-tokenizer -l en_PTB
```
```bash
echo 'Dont you wanna come?' | somajo-tokenizer -l en_PTB -
```
--------------------------------
### Tokenize XML Data (No Sentence Splitting)
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Tokenizes XML data without splitting sentences, treating the entire input as a single chunk. Prints the tokens of each chunk joined by spaces.
```python
>>> tokenizer = SoMaJo("de_CMC", split_sentences=False)
>>> chunks = tokenizer.tokenize_xml(xml, eos_tags)
>>> for chunk in chunks:
... print(" ".join([token.text for token in chunk]))
...
Heyi :)
Was machst du morgen Abend ?! Lust auf Film ? ;-)
```
--------------------------------
### Token Class
Source: https://context7.com/tsproisl/somajo/llms.txt
Details the structure and attributes of the `Token` object returned by SoMaJo.
```APIDOC
## Token Class
### Description
The `Token` class stores individual tokens with metadata including text, token class, spacing information, and character offsets.
### Attributes
- **text**: `str` - The token's text.
- **token_class**: `str` - The classification of the token (e.g., 'URL', 'date', 'regular').
- **space_after**: `bool` - Indicates if there is a space after the token.
- **original_spelling**: `str` - The original spelling of the token if different from `text` (e.g., after normalization).
- **extra_info**: `dict` - Additional information about the token.
- **character_offset**: `int` - The starting character offset of the token in the original text.
- **markup**: `str` - Any markup associated with the token (e.g., XML tags).
### Available Token Classes
'URL', 'XML_entity', 'XML_tag', 'abbreviation', 'action_word',
'amount', 'date', 'email_address', 'emoticon', 'hashtag',
'measurement', 'mention', 'number', 'ordinal', 'regular',
'semester', 'symbol', 'time'
### Request Example
```python
from somajo import SoMaJo
tokenizer = SoMaJo("de_CMC", character_offsets=True)
text = "Heyi:) Test!"
sentences = tokenizer.tokenize_text([text])
for sentence in sentences:
for token in sentence:
print(f"text: {token.text!r:10}")
print(f" token_class: {token.token_class}")
print(f" space_after: {token.space_after}")
print(f" original_spelling: {token.original_spelling}")
print(f" extra_info: {token.extra_info!r}")
print(f" character_offset: {token.character_offset}")
print(f" markup: {token.markup}")
print()
```
### Response Example
```
text: 'Heyi'
token_class: regular
space_after: True
original_spelling: Heyi
extra_info: {}
character_offset: 0
markup:
text: ':)'
token_class: emoticon
space_after: True
original_spelling: :)
extra_info: {}
character_offset: 4
markup:
text: 'Test!'
token_class: regular
space_after: False
original_spelling: Test!
extra_info: {}
character_offset: 7
markup:
```
```
--------------------------------
### Parallel Tokenization
Source: https://github.com/tsproisl/somajo/blob/master/README.md
Specify the number of worker processes for faster tokenization using the --parallel option.
```bash
somajo-tokenizer --parallel
```
--------------------------------
### Token Class Properties
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Details about the properties available for each Token object generated by SoMaJo.
```APIDOC
## Token Class
### Description
Represents a single token identified by the SoMaJo tokenizer.
### Properties
- **extra_info()** - Returns additional information about the token, such as its class and spacing.
```
--------------------------------
### Tokenize XML Data - Delimit Sentences with XML Tags
Source: https://github.com/tsproisl/somajo/blob/master/doc/source/somajo.md
Tokenizes XML data and delimits sentences using specified XML tags (e.g., ''). Prints each token on a new line, with an empty line after each delimited sentence.
```python
>>> xml = "Heyi:)
Was machst du morgen Abend?! Lust auf Film?;-)
"
>>> eos_tags = "title h1 h2 h3 h4 h5 h6 p br hr div ol ul dl table".split()
>>> tokenizer = SoMaJo("de_CMC", xml_sentences="s")
>>> sentences = tokenizer.tokenize_xml(xml, eos_tags)
>>> for sentence in sentences:
... for token in sentence:
... print(token.text)
... print()
...
Heyi
:)
Was
machst
du
morgen
Abend
?!
Lust
auf
Film
?
;-)
```
--------------------------------
### Tokenize XML File
Source: https://context7.com/tsproisl/somajo/llms.txt
Works like `tokenize_xml()` but reads from a file or file-like object. Can be used with `strip_tags` and `prune_tags`.
```python
from somajo import SoMaJo
tokenizer = SoMaJo("de_CMC")
eos_tags = ["title", "h1", "h2", "h3", "h4", "h5", "h6", "p", "br", "hr", "div"]
# From filename
sentences = tokenizer.tokenize_xml_file("document.html", eos_tags, strip_tags=True)
for sentence in sentences:
print(" ".join([token.text for token in sentence]))
# From file object with pruning
with open("document.html", encoding="utf-8") as f:
sentences = tokenizer.tokenize_xml_file(
f,
eos_tags,
strip_tags=True,
prune_tags=["script", "style"]
)
for sentence in sentences:
print(" ".join([token.text for token in sentence]))
```
--------------------------------
### Tokenize XML and strip tags, print sentences
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Tokenizes an XML file, strips all XML tags from the output, and prints each sentence on a single line. This is useful for extracting plain text content.
```python
>>> with open("example.xml") as f:
... sentences = tokenizer.tokenize_xml_file(f, eos_tags, strip_tags=True)
... for sentence in sentences:
... print(" ".join(token.text for token in sentence))
...
Heyi :)
Was machst du morgen Abend ? !
Lust auf Film ? ;-)
```
--------------------------------
### Tokenize Text File with Single Newlines
Source: https://github.com/tsproisl/somajo/blob/master/doc/source/somajo.md
Shows how to tokenize a text file where paragraphs are separated by single newlines. The output prints one sentence per line, joining tokens with spaces.
```APIDOC
## Tokenize Text File with Single Newlines
### Description
Tokenizes a text file with paragraphs separated by single newlines. Outputs each sentence on a new line, with tokens joined by spaces.
### Method
```python
tokenizer.tokenize_text_file()
```
### Endpoint
N/A (Library function)
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
```python
with open("example_single_newlines.txt", encoding="utf-8") as f:
print(f.read())
tokenizer = SoMaJo("de_CMC")
with open("example_empty_lines.txt", encoding="utf-8") as f:
sentences = tokenizer.tokenize_text_file(f, paragraph_separator="empty_lines")
for sentence in sentences:
print(" ".join([token.text for token in sentence]))
```
### Response
#### Success Response (200)
List of sentences, where each sentence is a string of tokens joined by spaces.
#### Response Example
```
Heyi :)
Was machst du morgen Abend ?!
Lust auf Film ? ;-)
```
```
--------------------------------
### CLI Parallel Processing for Large Files
Source: https://context7.com/tsproisl/somajo/llms.txt
Utilize parallel processing for large files directly from the command line to speed up tokenization.
```bash
# Parallel processing for large files
somajo-tokenizer --parallel 4 large_file.txt
```
--------------------------------
### Tokenize XML Data (Strip Tags)
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Tokenizes XML data and removes XML tags from the output. Prints one sentence per line, joining tokens with spaces.
```python
>>> sentences = tokenizer.tokenize_xml(xml, eos_tags, strip_tags=True)
>>> for sentence in sentences:
... print(" ".join([token.text for token in sentence]))
...
Heyi :)
Was machst du morgen Abend ?!
Lust auf Film ? ;-)
```
--------------------------------
### Tokenize Text File with Empty Lines
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Tokenizes a text file where paragraphs are separated by empty lines. Prints each token with its class and extra info, followed by an empty line per sentence.
```python
>>> with open("example_empty_lines.txt") as f:
... print(f.read())
...
Heyi:)
Was machst du morgen Abend?! Lust auf Film?;-)
>>> sentences = tokenizer.tokenize_text_file("example_empty_lines.txt", paragraph_separator="single_newlines")
>>> for sentence in sentences:
... for token in sentence:
... print("{token.text}\t{token.token_class}\t{token.extra_info}")
... print()
...
Heyi regular SpaceAfter=No
:) emoticon
Was regular
machst regular
du regular
morgen regular
Abend regular SpaceAfter=No
?! symbol
Lust regular
auf regular
Film regular SpaceAfter=No
? symbol SpaceAfter=No
;-) emoticon
```
--------------------------------
### Tokenize XML and Strip Tags
Source: https://github.com/tsproisl/somajo/blob/master/doc/source/somajo.md
Tokenizes an XML file, splits it into sentences, and removes all XML tags from the output. Prints each sentence as a single space-separated string. Accepts a file-like object for input.
```python
>>> with open("example.xml") as f:
... sentences = tokenizer.tokenize_xml_file(f, eos_tags, strip_tags=True)
... for sentence in sentences:
... print(" ".join(token.text for token in sentence))
...
Heyi :)
Was machst du morgen Abend ?!
Lust auf Film ? ;-)
```
--------------------------------
### Python: Tokenize XML Data
Source: https://github.com/tsproisl/somajo/blob/master/README.md
Process XML data using tokenize_xml_file or tokenize_xml. Specify end-of-sentence tags for sentence splitting.
```python
eos_tags = ["title", "h1", "p"]
# you can read from an open file object
sentences = tokenizer.tokenize_xml_file(file_object, eos_tags)
# or you can specify a file name
sentences = tokenizer.tokenize_xml_file("Beispieldatei.xml", eos_tags)
# or you can pass a string with XML data
sentences = tokenizer.tokenize_xml(xml_string, eos_tags)
for sentence in sentences:
for token in sentence:
print(token.text)
print()
```
--------------------------------
### Tokenize Text with Sentence Splitting Disabled
Source: https://context7.com/tsproisl/somajo/llms.txt
When `split_sentences=False` is set in the SoMaJo constructor, the `tokenize_text()` method yields complete tokenized paragraphs instead of individual sentences. This is useful when sentence boundaries are not required.
```python
from somajo import SoMaJo
tokenizer = SoMaJo("de_CMC", split_sentences=False)
paragraphs = ["Was machst du morgen Abend?! Lust auf Film?;-)"]
tokenized_paragraphs = tokenizer.tokenize_text(paragraphs)
for paragraph in tokenized_paragraphs:
print(" ".join([token.text for token in paragraph]))
```
--------------------------------
### Tokenize XML with Sentence Breaks on Specific Tags
Source: https://github.com/tsproisl/somajo/blob/master/README.md
For XML input, specify tags that always mark sentence breaks using multiple --tag options.
```bash
somajo-tokenizer --xml --split_sentences --tag h1 --tag p --tag div
```
--------------------------------
### English Tokenization
Source: https://context7.com/tsproisl/somajo/llms.txt
Use `language="en_PTB"` to tokenize English text following Penn Treebank conventions, which handles contractions and punctuation differently.
```python
from somajo import SoMaJo
tokenizer = SoMaJo(language="en_PTB")
paragraphs = ["Don't you wanna come? That ain't bad!:D"]
sentences = tokenizer.tokenize_text(paragraphs)
for sentence in sentences:
print(" ".join([token.text for token in sentence]))
# Output:
```
--------------------------------
### Tokenize text with camel case splitting
Source: https://github.com/tsproisl/somajo/blob/master/README.md
Tokenize a text file using EmpiriST 2015 guidelines, which includes splitting camel-cased tokens. Input is read from a file.
```bash
somajo-tokenizer -c
```
--------------------------------
### Tokenize Text with XML Sentence Delimiters
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Use this snippet to tokenize text and delimit sentences using XML tags. The output prints each token's text, enclosed by specified XML tags (e.g., and ).
```python
>>> tokenizer = SoMaJo("de_CMC", xml_sentences="s")
>>> sentences = tokenizer.tokenize_text(paragraphs)
>>> for sentence in sentences:
... for token in sentence:
... print(token.text)
...
...
```
--------------------------------
### Tokenize XML File
Source: https://context7.com/tsproisl/somajo/llms.txt
Reads XML or HTML content from a file or file-like object and tokenizes it, similar to `tokenize_xml()`.
```APIDOC
## tokenize_xml_file()
### Description
Works like `tokenize_xml()` but reads from a file or file-like object. Supports stripping and pruning of tags.
### Method
`tokenize_xml_file(source, eos_tags, strip_tags=False, prune_tags=None)`
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Parameters
- **source**: `str` or file-like object - The filename or file object to read XML/HTML from.
- **eos_tags**: `list[str]` - A list of tag names that mark sentence boundaries.
- **strip_tags**: `bool` (optional) - If `True`, XML tags are removed from the output.
- **prune_tags**: `list[str]` (optional) - A list of tag names whose content should be removed before tokenization.
### Request Example
```python
from somajo import SoMaJo
tokenizer = SoMaJo("de_CMC")
eos_tags = ["title", "h1", "h2", "h3", "h4", "h5", "h6", "p", "br", "hr", "div"]
# From filename
sentences = tokenizer.tokenize_xml_file("document.html", eos_tags, strip_tags=True)
for sentence in sentences:
print(" ".join([token.text for token in sentence]))
# From file object with pruning
with open("document.html", encoding="utf-8") as f:
sentences = tokenizer.tokenize_xml_file(
f,
eos_tags,
strip_tags=True,
prune_tags=["script", "style"]
)
for sentence in sentences:
print(" ".join([token.text for token in sentence]))
```
### Response
#### Success Response (200)
- **sentences**: `list[list[Token]]` or `list[list[str]]` - A list of sentences, tokenized from the XML file. The structure depends on the `strip_tags` parameter.
#### Response Example
```
This is important content .
This is also important content .
```
```
--------------------------------
### Tokenize Text with Sentence Splitting
Source: https://github.com/tsproisl/somajo/blob/master/doc/build/markdown/somajo.md
Use this snippet to tokenize paragraphs of text and split them into sentences. The output is a list of sentences, where each sentence is a list of Token objects. Each token's text is printed.
```python
>>> paragraphs = ["Heyi:)", "Was machst du morgen Abend?! Lust auf Film?;-)"]
>>> tokenizer = SoMaJo("de_CMC")
>>> sentences = tokenizer.tokenize_text(paragraphs)
>>> for sentence in sentences:
... print(" ".join([token.text for token in sentence]))
...
```
--------------------------------
### Tokenize English Text from Python
Source: https://github.com/tsproisl/somajo/blob/master/README.md
Tokenize English text using the SoMaJo tokenizer in Python. Specify the language as 'en_PTB' for Penn Treebank conventions.
```python
paragraphs = ["That aint bad!:D"]
tokenizer = SoMaJo(language="en_PTB")
sentences = tokenizer.tokenize_text(paragraphs)
```
--------------------------------
### Python: Tokenize File with Single Newline Paragraphs
Source: https://github.com/tsproisl/somajo/blob/master/README.md
Tokenize an entire file using the tokenize_text_file method, specifying that single newlines delimit paragraphs.
```python
sentences = tokenizer.tokenize_text_file("Beispieldatei.txt", paragraph_separator="single_newlines")
for sentence in sentences:
for token in sentence:
print(token.text)
print()
```