### Installation Guide

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Instructions for installing the benepar package, PyTorch, spaCy, and downloading pre-trained models.

```APIDOC
## Installation

Install the parser package and download pre-trained models for immediate use.

```bash
# Install the benepar package
pip install benepar

# Install PyTorch (required, GPU version recommended)
pip install torch

# Install spaCy and download English model for tokenization
pip install spacy
python -m spacy download en_core_web_md

# Download a benepar parsing model
python -c "import benepar; benepar.download('benepar_en3')"
```
```

--------------------------------

### Install benepar and dependencies

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Install the benepar package, PyTorch, spaCy, and download an English language model for spaCy. Ensure PyTorch is installed, preferably with GPU support.

```bash
pip install benepar

pip install torch

pip install spacy
python -m spacy download en_core_web_md

python -c "import benepar; benepar.download('benepar_en3')"
```

--------------------------------

### Install Berkeley Neural Parser

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Install the parser using pip. Version 0.2.0 is a major upgrade. Python 3.6+ and PyTorch 1.6+ are required.

```bash
pip install benepar
```

--------------------------------

### Install spaCy Model for English

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Install a spaCy model for English if using spaCy for tokenization and sentence segmentation. This command downloads the 'en_core_web_md' model.

```bash
python -m spacy download en_core_web_md
```

--------------------------------

### Download Parser Model

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Download a specific parser model, such as 'benepar_en3', after installing the library.

```python
import benepar
benepar.download('benepar_en3')
```

--------------------------------

### Usage with spaCy (spaCy v2)

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Integrate benepar with spaCy by loading a spaCy model and adding the BeneparComponent. This example is for spaCy version 2.

```python
import benepar, spacy
nlp = spacy.load('en_core_web_md')
if spacy.__version__.startswith('2'):
        nlp.add_pipe(benepar.BeneparComponent("benepar_en3"))
    else:
        nlp.add_pipe("benepar", config={"model": "benepar_en3"})
```

--------------------------------

### Download benepar parser models

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Download pre-trained parser models for various languages and accuracy levels. Models are stored locally and can be referenced by name. Examples include English, Chinese, German, and Arabic models.

```python
import benepar

# Download English model (T5-small based, 95.40 F1)
benepar.download('benepar_en3')

# Download high-accuracy English model (T5-large based, 96.29 F1)
benepar.download('benepar_en3_large')

# Download Chinese model (ELECTRA-based, 92.56 F1)
benepar.download('benepar_zh2')

# Download German model (XLM-R based, 92.10 F1)
benepar.download('benepar_de2')
```

--------------------------------

### Parse Sentence and Access Parse String

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

After setting up benepar with spaCy, process a document, get a sentence, and print its parse string. Access methods via Span._.

```python
doc = nlp("The time for action is now. It's never too late to do something.")
sent = list(doc.sents)[0]
print(sent._.parse_string)
```

--------------------------------

### Debug Raw Text Parsing (Discouraged)

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Example of parsing raw text directly, intended only for debugging or interactive use. For accurate parsing, use pre-tokenized input or spaCy integration.

```python
>>> parser.parse('"Fly safely."')  # For debugging/interactive use only.
```

--------------------------------

### Get Grammar Productions

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Obtain the grammar productions (rules) used to generate the parse tree. The `productions()` method returns a list of production objects.

```python
# Get grammar productions (rules)
productions = tree.productions()
for prod in productions[:5]:
    print(prod)
# TOP -> S
# S -> NP VP .
# NP -> DT NN
# DT -> 'The'
# NN -> 'cat'
```

--------------------------------

### Access Parse Children

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Iterate over the child spans of a given span in the parse tree. This example shows accessing children of the first sentence.

```python
list(sent._.children)[0]
```

--------------------------------

### Create benepar InputSentence with minimal specification

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Create an InputSentence with only the essential word list; the parser will infer spacing, tags, and escaped forms.

```python
import benepar

parser = benepar.Parser("benepar_en3")

# Minimal specification - parser guesses missing fields
simple_input = benepar.InputSentence(
    words=['This', 'is', 'a', 'test', '.']
)
tree = parser.parse(simple_input)
print(tree)
# Output: (TOP (S (NP (DT This)) (VP (VBZ is) (NP (DT a) (NN test))) (. .)))
```

--------------------------------

### BeneparComponent Configuration Options

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Demonstrates how to configure the BeneparComponent with custom token limits and part-of-speech tag handling.

```APIDOC
## BeneparComponent Configuration Options

Configure the BeneparComponent for optimal performance with custom token limits and part-of-speech tag handling.

```python
import benepar
import spacy

nlp = spacy.load('en_core_web_md')

# Example configuration (actual options not detailed in source text)
# nlp.add_pipe("benepar", config={"model": "benepar_en3", "max_tokens": 512, "use_pos": True})
```
```

--------------------------------

### Export Model for Distribution

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Convert training checkpoints to an optimized format for distribution and inference. Use the --compress flag to significantly reduce file size.

```bash
python src/export.py export \
    --model-path models/English_bert_large_dev=95.50.pt \
    --output-dir models/exported_english
```

```bash
python src/export.py export \
    --model-path models/English_bert_large_dev=95.50.pt \
    --output-dir models/exported_english_compressed \
    --compress \
    --test-path "data/wsj/test_23.LDC99T42"
```

```bash
python src/export.py test \
    --model-path models/exported_english \
    --test-path "data/wsj/test_23.LDC99T42"
```

--------------------------------

### spaCy Integration

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Demonstrates how to add the 'benepar' pipe to a spaCy pipeline with custom configurations and process documents to access parse strings.

```APIDOC
## Add benepar with custom configuration
nlp.add_pipe("benepar", config={
    "model": "benepar_en3",           # Model name or path to saved model
    "subbatch_max_tokens": 500,       # Max tokens per batch (adjust for GPU memory)
    "disable_tagger": False           # Set True to keep spaCy's POS tags
})

# Process documents
doc = nlp("The quick brown fox jumps over the lazy dog.")
sent = list(doc.sents)[0]
print(sent._.parse_string)
# Output: (S (NP (DT The) (JJ quick) (JJ brown) (NN fox)) (VP (VBZ jumps) (PP (IN over) (NP (DT the) (JJ lazy) (NN dog)))) (. .))

# Access token-level parse information
token = doc[0]  # "The"
print(f"Token '{token}' labels: {token._.labels}")  # ('DT',)
print(f"Token '{token}' parent: {token._.parent}" ) # The quick brown fox
```

--------------------------------

### Initialize benepar Parser with NLTK

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Initialize the standalone benepar parser for use with pre-tokenized data. This is suitable for treebanks or existing NLP pipelines.

```python
import benepar

# Initialize parser with model name
parser = benepar.Parser("benepar_en3")

# Parse a single sentence with full input specification
input_sentence = benepar.InputSentence(
    words=['"', 'Fly', 'safely', '.', '"'],
    space_after=[False, True, False, False, False],
    tags=['``', 'VB', 'RB', '.', "''"],
    escaped_words=['``', 'Fly', 'safely', '.', "''"],
)
tree = parser.parse(input_sentence)
print(tree)
# Output: (TOP (S (`` ``) (VP (VB Fly) (ADVP (RB safely))) (. .) ('' '')))

# Parse with minimal input (words only)
input_sentence = benepar.InputSentence(
    words=['The', 'time', 'for', 'action', 'is', 'now', '.'],
)
tree = parser.parse(input_sentence)
print(tree)
# Output: (TOP (S (NP (NP (DT The) (NN time)) (PP (IN for) (NP (NN action)))) (VP (VBZ is) (ADVP (RB now))) (. .)))

# Access tree structure using NLTK methods
print(f"Tree height: {tree.height()}")
print(f"Tree leaves: {tree.leaves()}")
print(f"Productions: {tree.productions()[:5]}")  # First 5 grammar rules
```

--------------------------------

### Create benepar InputSentence with full specification

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Define parser input with detailed tokenization information including original words, spacing, POS tags, and escaped representations for precise parse tree construction.

```python
import benepar

parser = benepar.Parser("benepar_en3")

# Full specification for precise control
input_sentence = benepar.InputSentence(
    # Original Unicode text for each token
    words=['"', 'Hello', ',', 'world', '!', '"'],
    # Whether whitespace follows each token
    space_after=[False, False, True, False, False, False],
    # Part-of-speech tags (optional, parser will predict if not provided)
    tags=['``', 'UH', ',', 'NN', '.', "''"],
    # How tokens appear in output tree (handles PTB escaping)
    escaped_words=['``', 'Hello', ',', 'world', '!', "''"],
)
tree = parser.parse(input_sentence)
print(tree)
# Output: (TOP (S (`` ``) (INTJ (UH Hello)) (, ,) (NP (NN world)) (. !) ('' '')))
```

--------------------------------

### Chinese Parsing with spaCy

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Set up benepar for Chinese parsing by loading a Chinese spaCy model and specifying the 'benepar_zh2' model.

```python
# Chinese parsing
nlp_zh = spacy.load('zh_core_web_md')
nlp_zh.add_pipe("benepar", config={"model": "benepar_zh2"})
doc = nlp_zh("今天天气很好。")
for sent in doc.sents:
    print(sent._.parse_string)
```

--------------------------------

### benepar.Parser (NLTK Integration)

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Shows how to use the benepar.Parser class with pre-tokenized data and NLTK, suitable for processing treebanks or integrating with existing NLP pipelines.

```APIDOC
## benepar.Parser (NLTK Integration)

import benepar

# Initialize parser with model name
parser = benepar.Parser("benepar_en3")

# Parse a single sentence with full input specification
input_sentence = benepar.InputSentence(
    words=['"', 'Fly', 'safely', '.', '"'],
    space_after=[False, True, False, False, False],
    tags=['``', 'VB', 'RB', '.', "''"],
    escaped_words=['``', 'Fly', 'safely', '.', "''"],
)
tree = parser.parse(input_sentence)
print(tree)
# Output: (TOP (S (`` ``) (VP (VB Fly) (ADVP (RB safely))) (. .) ('' '')))

# Parse with minimal input (words only)
input_sentence = benepar.InputSentence(
    words=['The', 'time', 'for', 'action', 'is', 'now', '.'])
tree = parser.parse(input_sentence)
print(tree)
# Output: (TOP (S (NP (NP (DT The) (NN time)) (PP (IN for) (NP (NN action)))) (VP (VBZ is) (ADVP (RB now))) (. .)))

# Access tree structure using NLTK methods
print(f"Tree height: {tree.height()}")
print(f"Tree leaves: {tree.leaves()}")
print(f"Productions: {tree.productions()[:5]}")  # First 5 grammar rules
```

--------------------------------

### Export and Compress Model for Distribution

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

This command exports a model checkpoint and applies compression to reduce the file size. It's recommended to use the `--test-path` option to verify that compression does not significantly impact parsing accuracy.

```bash
python src/export.py export \
  --model-path models/en_bert_base_dev=*.pt \
  --output-dir=models/en_bert_base \
  --test-path=data/wsj/test_23.LDC99T42
```

--------------------------------

### benepar.download() - Model Download Function

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Function to download pre-trained parser models. Models are stored locally and can be referenced by name.

```APIDOC
## benepar.download()

Downloads pre-trained parser models from the benepar server. Models are stored locally and can be referenced by name in subsequent parsing operations. Available models include language-specific variants for 11 languages with varying accuracy/speed tradeoffs.

```python
import benepar

# Download English model (T5-small based, 95.40 F1)
benepar.download('benepar_en3')

# Download high-accuracy English model (T5-large based, 96.29 F1)
benepar.download('benepar_en3_large')

# Download Chinese model (ELECTRA-based, 92.56 F1)
benepar.download('benepar_zh2')

# Download German model (XLM-R based, 92.10 F1)
benepar.download('benepar_de2')

# Available models:
# - benepar_en3: English (recommended)
# - benepar_en3_large: English high-accuracy
# - benepar_en3_wsj: English (canonical WSJ tokenization)
# - benepar_zh2: Chinese
# - benepar_ar2: Arabic
# - benepar_de2: German
# - benepar_eu2: Basque
# - benepar_fr2: French
# - benepar_he2: Hebrew
# - benepar_hu2: Hungarian
# - benepar_ko2: Korean
# - benepar_pl2: Polish
# - benepar_sv2: Swedish
```
```

--------------------------------

### Train English Constituency Parser with BERT

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Basic command-line training for an English constituency parser using a pre-trained BERT encoder. Requires treebank data in bracketed format.

```bash
python src/main.py train \
    --train-path "data/wsj/train_02-21.LDC99T42" \
    --dev-path "data/wsj/dev_22.LDC99T42" \
    --use-pretrained --pretrained-model "bert-large-uncased" \
    --use-encoder --num-layers 2 \
    --predict-tags \
    --model-path-base models/English_bert_large
```

--------------------------------

### German Parsing with spaCy

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Configure benepar for German parsing by loading a German spaCy model and adding the 'benepar' pipe with the appropriate model.

```python
import benepar
import spacy

# German parsing
nlp_de = spacy.load('de_core_news_md')
nlp_de.add_pipe("benepar", config={"model": "benepar_de2"})
doc = nlp_de("Der schnelle braune Fuchs springt über den faulen Hund.")
for sent in doc.sents:
    print(sent._.parse_string)
```

--------------------------------

### Add benepar with custom configuration to spaCy

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Integrate the benepar parser into a spaCy pipeline with custom model and processing configurations. Adjust `subbatch_max_tokens` for GPU memory.

```python
import spacy

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("benepar", config={
    "model": "benepar_en3",           # Model name or path to saved model
    "subbatch_max_tokens": 500,       # Max tokens per batch (adjust for GPU memory)
    "disable_tagger": False           # Set True to keep spaCy's POS tags
})

# Process documents
doc = nlp("The quick brown fox jumps over the lazy dog.")
sent = list(doc.sents)[0]
print(sent._.parse_string)
# Output: (S (NP (DT The) (JJ quick) (JJ brown) (NN fox)) (VP (VBZ jumps) (PP (IN over) (NP (DT the) (JJ lazy) (NN dog)))) (. .))

# Access token-level parse information
token = doc[0]  # "The"
print(f"Token '{token}' labels: {token._.labels}")  # ('DT',)
print(f"Token '{token}' parent: {token._.parent}")  # The quick brown fox
```

--------------------------------

### Evaluate English Parser and Save Predictions

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Evaluate an English constituency parser and save the predictions to a file. Use the --output-path argument to specify the output file.

```bash
python src/main.py test \
    --model-path models/English_bert_large_dev=95.50.pt \
    --test-path "data/wsj/test_23.LDC99T42" \
    --output-path predictions.txt \
    --no-predict-tags
```

--------------------------------

### Train SPMRL Language Parser with Multilingual BERT

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Command-line training for SPMRL languages using a multilingual BERT model. Set the SPMRL_LANG environment variable and specify the appropriate paths.

```bash
SPMRL_LANG=German
python src/main.py train \
    --train-path data/spmrl/${SPMRL_LANG}.train \
    --dev-path data/spmrl/${SPMRL_LANG}.dev \
    --evalb-dir EVALB_SPMRL \
    --use-pretrained --pretrained-model "bert-base-multilingual-cased" \
    --use-encoder --num-layers 2 \
    --predict-tags \
    --model-path-base models/${SPMRL_LANG}_bert_multilingual
```

--------------------------------

### Export Model Checkpoint to Directory

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Use this command to convert a saved checkpoint file into a directory containing all necessary components for inference. This process is useful for creating distributable model packages.

```bash
python src/export.py export \
  --model-path models/en_bert_base_dev=*.pt \
  --output-dir=models/en_bert_base
```

--------------------------------

### Configure BeneparComponent in spaCy

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Configure the BeneparComponent for custom token limits and part-of-speech tag handling when integrating with spaCy. This allows for fine-tuning the parsing process.

```python
import benepar
import spacy

nlp = spacy.load('en_core_web_md')

```

--------------------------------

### Train Chinese Constituency Parser with BERT

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Command-line training for a Chinese constituency parser using a pre-trained Chinese BERT model. Specify 'chinese' for text processing.

```bash
python src/main.py train \
    --train-path "data/ctb_5.1/ctb.train" \
    --dev-path "data/ctb_5.1/ctb.dev" \
    --text-processing "chinese" \
    --use-pretrained --pretrained-model "bert-base-chinese" \
    --predict-tags \
    --model-path-base models/Chinese_bert
```

--------------------------------

### Train a Self-Attentive Parser

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Use this command to train a new parser model. Specify `--use-pretrained` to leverage pre-trained encoders and `--model-path-base` to define where trained models will be saved.

```bash
python src/main.py train --use-pretrained --model-path-base models/en_bert_base
```

--------------------------------

### Create benepar InputSentence using escaped_words

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Instantiate an InputSentence using only `escaped_words`, useful when processing data that already conforms to treebank escaping conventions.

```python
import benepar

parser = benepar.Parser("benepar_en3")

# Using escaped_words only (e.g., from treebank data)
treebank_input = benepar.InputSentence(
    escaped_words=['-LRB-', 'parenthetical', '-RRB-']
)
tree = parser.parse(treebank_input)
print(tree)
```

--------------------------------

### Train English Model With BERT Pre-training

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/EXPERIMENTS.md

Train an English parser using a pre-trained BERT model (bert-large-uncased). This command enables the encoder and specifies the number of layers. Ensure the model path base is set correctly.

```bash
python src/main.py train \
    --train-path "data/wsj/train_02-21.LDC99T42" \
    --dev-path "data/wsj/dev_22.LDC99T42" \
    --use-pretrained --pretrained-model "bert-large-uncased" \
    --use-encoder --num-layers 2 \
    --predict-tags \
    --model-path-base models/English_bert_large_uncased
```

--------------------------------

### Test Exported English Parser

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Verify the performance of an exported English parser using a specified test set. This command uses the `test` subcommand of `src/export.py` and is suitable for models that have been exported.

```bash
python src/export.py test --model-path benepar_en3_wsj --test-path data/wsj/test_23.LDC99T42
```

--------------------------------

### French Parsing with spaCy

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Enable French parsing by loading a French spaCy model and configuring the 'benepar' pipe with the 'benepar_fr2' model.

```python
# French parsing
nlp_fr = spacy.load('fr_core_news_md')
nlp_fr.add_pipe("benepar", config={"model": "benepar_fr2"})
doc = nlp_fr("Le chat est sur le tapis.")
for sent in doc.sents:
    print(sent._.parse_string)
```

--------------------------------

### Parse Sentence with NLTK Interface

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Parse a sentence using the benepar parser and the NLTK interface. Requires initializing the parser with a model path.

```python
import benepar

parser = benepar.Parser("benepar_en3")
tree = parser.parse(benepar.InputSentence(
    words=['The', 'cat', 'sat', 'on', 'the', 'mat', '.']
))
```

--------------------------------

### Parse Single Sentence with NLTK

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Demonstrates parsing a single sentence using the NLTK interface with a pre-tokenized `InputSentence` object. Ensure all required fields for `InputSentence` are provided for optimal results.

```python
>>> import benepar
>>> parser = benepar.Parser("benepar_en3")
>>> input_sentence = benepar.InputSentence(
    words=['"', 'Fly', 'safely', '.', '"'],
    space_after=[False, True, False, False, False],
    tags=['``', 'VB', 'RB', '.', "''"],
    escaped_words=['``', 'Fly', 'safely', '.', "''"],
)
>>> tree = parser.parse(input_sentence)
>>> print(tree)
(TOP (S (`` ``) (VP (VB Fly) (ADVP (RB safely))) (. .) ('' '')))
```

--------------------------------

### NLTK Interface for Arabic and Hebrew

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Use the NLTK interface for languages not supported by spaCy, such as Arabic and Hebrew, by initializing the benepar.Parser directly with the language model.

```python
# Languages requiring NLTK interface only (no spaCy support):
# - Arabic (benepar_ar2)
# - Hebrew (benepar_he2)
parser_ar = benepar.Parser("benepar_ar2")
parser_he = benepar.Parser("benepar_he2")
# Use with pre-tokenized InputSentence
```

--------------------------------

### Train Chinese Model with BERT

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/EXPERIMENTS.md

Train a Chinese parser using a pre-trained Chinese BERT model. This command specifies the text processing mode for Chinese and enables tag prediction. Ensure the correct paths for training and development data are provided.

```bash
python src/main.py train \
    --train-path "data/ctb_5.1/ctb.train" \
    --dev-path "data/ctb_5.1/ctb.dev" \
    --text-processing "chinese" \
    --use-pretrained --pretrained-model "bert-base-chinese" \
    --predict-tags \
    --model-path-base models/Chinese_bert_base_chinese
```

--------------------------------

### benepar.InputSentence

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Details the benepar.InputSentence data class for specifying parser input with tokenization details, offering control over word representation, escaping, and spacing.

```APIDOC
## benepar.InputSentence

import benepar

parser = benepar.Parser("benepar_en3")

# Full specification for precise control
input_sentence = benepar.InputSentence(
    # Original Unicode text for each token
    words=['"', 'Hello', ',', 'world', '!', '"'],
    # Whether whitespace follows each token
    space_after=[False, False, True, False, False, False],
    # Part-of-speech tags (optional, parser will predict if not provided)
    tags=['``', 'UH', ',', 'NN', '.', "''"],
    # How tokens appear in output tree (handles PTB escaping)
    escaped_words=['``', 'Hello', ',', 'world', '!', "''"],
)
tree = parser.parse(input_sentence)
print(tree)
# Output: (TOP (S (`` ``) (INTJ (UH Hello)) (, ,) (NP (NN world)) (. !) ('' '')))

# Minimal specification - parser guesses missing fields
simple_input = benepar.InputSentence(
    words=['This', 'is', 'a', 'test', '.'])
tree = parser.parse(simple_input)
print(tree)
# Output: (TOP (S (NP (DT This)) (VP (VBZ is) (NP (DT a) (NN test))) (. .)))

# Using escaped_words only (e.g., from treebank data)
treebank_input = benepar.InputSentence(
    escaped_words=['-LRB-', 'parenthetical', '-RRB-'])
tree = parser.parse(treebank_input)
print(tree)
```

--------------------------------

### Parse Multiple Sentences with NLTK

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Illustrates how to parse multiple sentences efficiently using the `parse_sents` method with a list of `InputSentence` objects.

```python
>>> input_sentence1 = benepar.InputSentence(
    words=['The', 'time', 'for', 'action', 'is', 'now', '.'],
)
>>> input_sentence2 = benepar.InputSentence(
    words=['It', "'", 's', 'never', 'too', 'late', 'to', 'do', 'something', '.'],
)
>>> parser.parse_sents([input_sentence1, input_sentence2])
```

--------------------------------

### Accessing spaCy Extension Attributes with Benepar

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Demonstrates how to use Benepar's extension attributes on spaCy Span and Token objects to retrieve constituency labels, parse trees, and hierarchical relationships. Ensure Benepar is added as a pipe to the spaCy pipeline.

```python
import benepar
import spacy

lp = spacy.load('en_core_web_md')
nlp.add_pipe("benepar", config={"model": "benepar_en3"})

doc = nlp("The quick brown fox jumps over the lazy dog.")
sent = list(doc.sents)[0]

# Span._.labels - tuple of constituency labels for this span
# Spans can have multiple labels for unary chains
print(f"Sentence labels: {sent._.labels}")  # ('S',)

# Span._.parse_string - bracketed parse tree string
print(f"Parse string: {sent._.parse_string}")
# (S (NP (DT The) (JJ quick) (JJ brown) (NN fox)) (VP (VBZ jumps) (PP (IN over) (NP (DT the) (JJ lazy) (NN dog)))) (. .))

# Span._.constituents - iterator over all sub-constituents (pre-order)
print("All constituents:")
for const in sent._.constituents:
    print(f"  [{const.start}:{const.end}] '{const}' -> {const._.labels}")

# Span._.children - iterator over direct child spans
print("Direct children of sentence:")
for child in sent._.children:
    print(f"  '{child}' -> {child._.labels}")
# Output:
#   'The quick brown fox' -> ('NP',)
#   'jumps over the lazy dog' -> ('VP',)
#   '.' -> ('.',)

# Span._.parent - parent span in the parse tree (None for root)
np_span = doc[0:4]  # "The quick brown fox"
print(f"Parent of NP: {np_span._.parent}")  # Full sentence

# Token extensions (operate on single-token spans)
token = doc[3]  # "fox"
print(f"Token '{token}' labels: {token._.labels}")  # ('NN',)
print(f"Token '{token}' parent: {token._.parent}")  # The quick brown fox

# Handle NonConstituentException for invalid spans
from benepar import NonConstituentException
try:
    invalid_span = doc[1:3]  # "quick brown" - not a constituent
    labels = invalid_span._.labels
except NonConstituentException as e:
    print(f"Not a valid constituent: {invalid_span}")
```

--------------------------------

### Train English Model Without Pre-training

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/EXPERIMENTS.md

Use this command to train an English parser using character LSTMs and an encoder, without leveraging pre-trained models. Adjust batch size and learning rate as needed.

```bash
python src/main.py train \
    --train-path "data/wsj/train_02-21.LDC99T42" \
    --dev-path "data/wsj/dev_22.LDC99T42" \
    --use-chars-lstm --use-encoder --num-layers 8 \
    --batch-size 250 --learning-rate 0.0008 \
    --model-path-base models/English_charlstm
```

--------------------------------

### Evaluate a Trained Self-Attentive Parser

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Execute this command to evaluate a previously trained parser model on a test corpus. Ensure `--model-path` points to the saved model file.

```bash
python src/main.py test --model-path models/en_bert_base_dev=*.pt
```

--------------------------------

### benepar.BeneparComponent - spaCy Integration

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Integrates benepar into spaCy's NLP pipeline, adding constituency parsing capabilities to spaCy documents.

```APIDOC
## benepar.BeneparComponent (spaCy Integration)

The recommended way to use benepar through spaCy's NLP pipeline. The component adds constituency parsing capabilities to spaCy documents, providing access to parse trees through extension attributes on Span and Token objects.

```python
import benepar
import spacy

# Load spaCy model and add benepar component
nlp = spacy.load('en_core_web_md')

# For spaCy 3.x
nlp.add_pipe("benepar", config={"model": "benepar_en3"})

# For spaCy 2.x (alternative syntax)
# nlp.add_pipe(benepar.BeneparComponent("benepar_en3"))

# Parse a document with multiple sentences
doc = nlp("The time for action is now. It's never too late to do something.")

# Iterate over sentences and access parse information
for sent in doc.sents:
    # Get the full parse tree as a string
    print(f"Parse tree: {sent._.parse_string}")
    # Output: (S (NP (NP (DT The) (NN time)) (PP (IN for) (NP (NN action)))) (VP (VBZ is) (ADVP (RB now))) (. .))

    # Get constituency labels for the sentence span
    print(f"Labels: {sent._.labels}")
    # Output: ('S',)

    # Iterate over child constituents
    for child in sent._.children:
        print(f"Child: '{child}' with labels {child._.labels}")
    # Output:
    # Child: 'The time for action' with labels ('NP',)
    # Child: 'is now' with labels ('VP',)
    # Child: '.' with labels ('.',)

# Access constituents for a specific span
span = doc[0:4]  # "The time for action"
print(f"Span labels: {span._.labels}")  # ('NP',)

# Get parent constituent
child_span = doc[0:2]  # "The time"
parent = child_span._.parent
print(f"Parent of '{child_span}': '{parent}'")  # Parent of 'The time': 'The time for action'

# Iterate over all sub-constituents in pre-order traversal
for constituent in sent._.constituents:
    print(f"Constituent: {constituent} -> {constituent._.labels}")
```
```

--------------------------------

### Batch parse sentences with benepar.Parser.parse_sents()

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Efficiently parse multiple sentences using `parse_sents`, which supports batching for optimal GPU utilization. Input can be `InputSentence` objects or lists of words.

```python
import benepar

parser = benepar.Parser("benepar_en3", batch_size=64)

# Parse multiple sentences as InputSentence objects
sentences = [
    benepar.InputSentence(words=['The', 'cat', 'sat', '.']),
    benepar.InputSentence(words=['The', 'dog', 'ran', '.']),
    benepar.InputSentence(words=['Birds', 'fly', 'south', '.']),
]

# Returns a generator - iterate to get results
for tree in parser.parse_sents(sentences):
    print(tree)
# Output:
# (TOP (S (NP (DT The) (NN cat)) (VP (VBD sat)) (. .)))
# (TOP (S (NP (DT The) (NN dog)) (VP (VBD ran)) (. .)))
# (TOP (S (NP (NNS Birds)) (VP (VBP fly) (ADVP (RB south))) (. .)))
```

```python
# Parse from list of word lists
word_lists = [
    ['I', 'love', 'parsing', '.'],
    ['NLP', 'is', 'fun', '!'],
]
for tree in parser.parse_sents(word_lists):
    print(tree.pformat(margin=200))  # Pretty print on one line
```

```python
# Parse raw text (for debugging only - spaCy recommended for production)
# Parser will use NLTK for sentence splitting and tokenization
trees = list(parser.parse_sents("Hello world. How are you?"))
for tree in trees:
    print(tree)
```

--------------------------------

### Evaluate English Constituency Parser

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Evaluate parser performance on an English WSJ test set using the EVALB metric. Ensure the model path points to a trained checkpoint.

```bash
python src/main.py test \
    --model-path models/English_bert_large_dev=95.50.pt \
    --test-path "data/wsj/test_23.LDC99T42" \
    --no-predict-tags
```

--------------------------------

### Evaluate English BERT Model

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/EXPERIMENTS.md

Evaluate a trained English BERT model. This command requires the path to the test set and the trained model file. Tags are not predicted during evaluation.

```bash
python src/main.py test \
    --test-path "data/wsj/test_23.LDC99T42" \
    --no-predict-tags \
    --model-path models/English_bert_large_uncased_*.pt
```

--------------------------------

### Integrate benepar with spaCy

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Add the benepar component to a spaCy NLP pipeline for constituency parsing. This enables access to parse trees and labels via extension attributes on spaCy document objects. Supports spaCy 3.x and 2.x.

```python
import benepar
import spacy

# Load spaCy model and add benepar component
nlp = spacy.load('en_core_web_md')

# For spaCy 3.x
nlp.add_pipe("benepar", config={"model": "benepar_en3"})

# For spaCy 2.x (alternative syntax)
# nlp.add_pipe(benepar.BeneparComponent("benepar_en3"))

# Parse a document with multiple sentences
doc = nlp("The time for action is now. It's never too late to do something.")

# Iterate over sentences and access parse information
for sent in doc.sents:
    # Get the full parse tree as a string
    print(f"Parse tree: {sent._.parse_string}")
    # Output: (S (NP (NP (DT The) (NN time)) (PP (IN for) (NP (NN action)))) (VP (VBZ is) (ADVP (RB now))) (. .))

    # Get constituency labels for the sentence span
    print(f"Labels: {sent._.labels}")
    # Output: ('S',)

    # Iterate over child constituents
    for child in sent._.children:
        print(f"Child: '{child}' with labels {child._.labels}")
    # Output:
    # Child: 'The time for action' with labels ('NP',)
    # Child: 'is now' with labels ('VP',)
    # Child: '.' with labels ('.',)

# Access constituents for a specific span
span = doc[0:4]  # "The time for action"
print(f"Span labels: {span._.labels}")  # ('NP',)

# Get parent constituent
child_span = doc[0:2]  # "The time"
parent = child_span._.parent
print(f"Parent of '{child_span}': '{parent}'")  # Parent of 'The time': 'The time for action'

# Iterate over all sub-constituents in pre-order traversal
for constituent in sent._.constituents:
    print(f"Constituent: {constituent} -> {constituent._.labels}")
```

--------------------------------

### Parser.parse_sents()

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Explains the batch parsing method `parse_sents()` for efficiently processing multiple sentences, optimizing GPU utilization and memory.

```APIDOC
## Parser.parse_sents()

import benepar

parser = benepar.Parser("benepar_en3", batch_size=64)

# Parse multiple sentences as InputSentence objects
sentences = [
    benepar.InputSentence(words=['The', 'cat', 'sat', '.']),
    benepar.InputSentence(words=['The', 'dog', 'ran', '.']),
    benepar.InputSentence(words=['Birds', 'fly', 'south', '.']),
]

# Returns a generator - iterate to get results
for tree in parser.parse_sents(sentences):
    print(tree)
# Output:
# (TOP (S (NP (DT The) (NN cat)) (VP (VBD sat)) (. .)))
# (TOP (S (NP (DT The) (NN dog)) (VP (VBD ran)) (. .)))
# (TOP (S (NP (NNS Birds)) (VP (VBP fly) (ADVP (RB south))) (. .)))

# Parse from list of word lists
word_lists = [
    ['I', 'love', 'parsing', '.'],
    ['NLP', 'is', 'fun', '!'],
]
for tree in parser.parse_sents(word_lists):
    print(tree.pformat(margin=200))  # Pretty print on one line

# Parse raw text (for debugging only - spaCy recommended for production)
# Parser will use NLTK for sentence splitting and tokenization
trees = list(parser.parse_sents("Hello world. How are you?"))
for tree in trees:
    print(tree)
```

--------------------------------

### Train SPMRL Model with BERT

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/EXPERIMENTS.md

Train an SPMRL parser using a multilingual BERT model. This command includes conditional arguments for Arabic and Hebrew text processing and maximum sequence lengths. Ensure the correct language is set for SPMRL_LANG.

```bash
SPMRL_LANG=Arabic
echo "Language: ${SPMRL_LANG}"

EXTRA_ARGS=
if [ "$SPMRL_LANG" = "Arabic" ]; then
    # There are sentences in the train and dev sets that are too long for BERT.
    # Fortunately, there are no such long sentences in the test set
    EXTRA_ARGS="--text-processing arabic-translit --max-len-train 266 --max-len-dev 494"
fi
if [ "$SPMRL_LANG" = "Hebrew" ]; then
    EXTRA_ARGS="--text-processing hebrew"
fi

python src/main.py train \
    --train-path data/spmrl/${SPMRL_LANG}.train \
    --dev-path data/spmrl/${SPMRL_LANG}.dev \
    --evalb-dir EVALB_SPMRL \
    --use-pretrained --pretrained-model "bert-base-multilingual-cased" \
    --use-encoder --num-layers 2 \
    --predict-tags \
    --model-path-base models/${SPMRL_LANG}_bert_base_multilingual_cased \
    $EXTRA_ARGS
```

--------------------------------

### Evaluate SPMRL Language Parser

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Evaluate a constituency parser trained for an SPMRL language. Provide the model path, test path, and EVALB directory.

```bash
python src/main.py test \
    --model-path models/German_bert_multilingual_dev=91.50.pt \
    --test-path data/spmrl/German.test \
    --evalb-dir EVALB_SPMRL \
    --no-predict-tags
```

--------------------------------

### Parse Single Sentence with Minimal Input

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

Shows parsing a single sentence with `InputSentence` when only the `words` field is provided. The parser attempts to infer missing fields.

```python
>>> input_sentence = benepar.InputSentence(
    words=['"', 'Fly', 'safely', '.', '"'],
)
>>> parser.parse(input_sentence)
```

--------------------------------

### Evaluate Chinese BERT Model

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/EXPERIMENTS.md

Evaluate a trained Chinese BERT parser. This command requires the path to the test set and the trained model file, along with the Chinese text processing mode. Tags are not predicted during evaluation.

```bash
python src/main.py test \
    --test-path "data/ctb_5.1/ctb.test" \
    --text-processing "chinese" \
    --no-predict-tags \
    --model-path models/Chinese_bert_base_chinese_*.pt
```

--------------------------------

### Convert Tree to String Format

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Format the parse tree into a single-line string representation using `pformat`. The `margin` parameter controls line wrapping.

```python
# Convert to other formats
print(tree.pformat(margin=200))  # Single line format
# tree.draw()  # GUI visualization (requires Tkinter)
```

--------------------------------

### Evaluate Chinese Constituency Parser

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Evaluate a trained Chinese constituency parser on its test set. Specify 'chinese' for text processing and provide the correct model and test paths.

```bash
python src/main.py test \
    --model-path models/Chinese_bert_dev=92.00.pt \
    --test-path "data/ctb_5.1/ctb.test" \
    --text-processing "chinese" \
    --no-predict-tags
```

--------------------------------

### Multilingual spaCy Model for Multiple Languages

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Utilize the 'xx_sent_ud_sm' spaCy model for various languages, including Korean, Polish, and Swedish, with benepar. Ensure spaCy 3.0+ is used.

```python
# Using multilingual spaCy model for multiple languages
# (Korean, Polish, Swedish work with xx_sent_ud_sm in spaCy 3.0+)
nlp_multi = spacy.load('xx_sent_ud_sm')
nlp_multi.add_pipe("benepar", config={"model": "benepar_ko2"})
# Note: xx_sent_ud_sm handles tokenization for these languages
```

--------------------------------

### Navigate Tree Structure

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Traverse the parse tree by accessing its children nodes. Nodes can be indexed to retrieve specific subtrees.

```python
# Navigate tree structure
s_node = tree[0]  # First child (S node)
print(f"S node: {s_node.label()}")  # S

np_node = s_node[0]  # First child of S (NP)
print(f"NP subtree: {np_node}")  # (NP (DT The) (NN cat))
```

--------------------------------

### Citation for Multilingual Constituency Parsing Paper

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/README.md

BibTeX entry for citing the paper 'Multilingual Constituency Parsing with Self-Attention and Pre-Training' (ACL 2019).

```bibtex
@inproceedings{kitaev-etal-2019-multilingual,
    title = "Multilingual Constituency Parsing with Self-Attention and Pre-Training",
    author = "Kitaev, Nikita  and
      Cao, Steven  and
      Klein, Dan",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/P19-1340",
    doi = "10.18653/v1/P19-1340",
    pages = "3499--3505",
}

```

--------------------------------

### Evaluate SPMRL Model with BERT

Source: https://github.com/nikitakit/self-attentive-parser/blob/master/EXPERIMENTS.md

Evaluate a trained SPMRL parser using a multilingual BERT model. This command includes conditional arguments for text processing based on the language. Ensure the correct language is set for SPMRL_LANG.

```bash
SPMRL_LANG=Arabic
echo "Language: ${SPMRL_LANG}"

EXTRA_ARGS=
if [ "$SPMRL_LANG" = "Arabic" ]; then
    EXTRA_ARGS="--text-processing arabic-translit"
fi
if [ "$SPMRL_LANG" = "Hebrew" ]; then
    EXTRA_ARGS="--text-processing hebrew"
fi

python src/main.py test \
    --test-path data/spmrl/${SPMRL_LANG}.test \
    --evalb-dir EVALB_SPMRL \
    --no-predict-tags \
    --model-path models/${SPMRL_LANG}_bert_base_multilingual_cased_*.pt \
    $EXTRA_ARGS
```

--------------------------------

### Use Exported Model in Python

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Load and use an exported model directly in Python scripts or integrate it with spaCy pipelines.

```python
import benepar
parser = benepar.Parser("models/exported_english")
```

```python
# Or use directly in spaCy
nlp.add_pipe("benepar", config={"model": "models/exported_english"})
```

--------------------------------

### Basic Tree Properties

Source: https://context7.com/nikitakit/self-attentive-parser/llms.txt

Access fundamental properties of a parse tree, such as its root label, height, and leaf nodes.

```python
# Basic tree properties
print(f"Root label: {tree.label()}")  # TOP
print(f"Height: {tree.height()}")  # 7
print(f"Leaves: {tree.leaves()}")  # ['The', 'cat', 'sat', 'on', 'the', 'mat', '.']
```