Stanza

https://stanfordnlp.github.io/stanza/
Admin
Stanza is a Python natural language processing package that provides accurate and efficient tools...
Tokens:60,683
Snippets:756
Trust Score:8.3
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install Stanza from Source

Source: https://stanfordnlp.github.io/stanza/installation_usage

Install Stanza directly from its git repository for development purposes. Ensure Cuda and PyTorch are installed first.

```bash
git clone https://github.com/stanfordnlp/stanza.git
cd stanza
pip install -e .

```

--------------------------------

### Install WikiExtractor fork

Source: https://stanfordnlp.github.io/stanza/new_language_charlm.html

Install the recommended fork of WikiExtractor to avoid known issues with template self-inclusion loops.

```bash
pip install git+https://github.com/AngledLuffa/wikiextractor.git
```

--------------------------------

### Install CoreNLP to a Custom Directory

Source: https://stanfordnlp.github.io/stanza/client_setup

Installs CoreNLP to a specified directory. Ensure the `$CORENLP_HOME` environment variable points to this location after installation.

```python
import stanza
stanza.install_corenlp(dir="YOUR_CORENLP_FOLDER")
```

--------------------------------

### Install Dependencies for Similarity Inspection

Source: https://stanfordnlp.github.io/stanza/new_language_charlm.html

Install the required packages for inspecting document similarity.

```bash
pip install datasketch py-tlsh zstandard  # py-tlsh optional but recommended
```

--------------------------------

### Train Coreference Model with Weights and Biases

Source: https://stanfordnlp.github.io/stanza/new_language_coref.html

Starts the coreference model training process with experiment tracking enabled via Weights and Biases. Ensure W&B is installed and configured.

```bash
python -m stanza.models.wl_coref train [your experiment] --config_file ./data/coref_config.toml --wandb

```

--------------------------------

### Token Information Example Output

Source: https://stanfordnlp.github.io/stanza/client_usage.html

Example output showing the token value, its Part-of-Speech tag, and Named Entity Recognition tag.

```text
Chris NNP PERSON

```

--------------------------------

### Start CoreNLP Client and Annotate Text

Source: https://stanfordnlp.github.io/stanza/client_usage.html

Instantiate the CoreNLPClient within a context manager to ensure proper shutdown. Pass the text and desired annotators to the client. The server starts automatically.

```python
text = "Chris Manning is a nice person. Chris wrote a simple sentence. He also gives oranges to people."
with CoreNLPClient(
        annotators=["tokenize","ssplit","pos","lemma","ner", "parse", "depparse","corefකිය"],
        timeout=30000,
        memory='6G') as client:
    ann = client.annotate(text)

```

--------------------------------

### Install Stanza via pip

Source: https://stanfordnlp.github.io/stanza/installation_usage

Use this command to install the latest version of Stanza. It also installs necessary dependencies like PyTorch.

```bash
pip install stanza

```

--------------------------------

### Launch Server with French Defaults

Source: https://stanfordnlp.github.io/stanza/client_properties.html

Start a CoreNLP server pre-configured with French language defaults. This simplifies processing French text.

```python
with CoreNLPClient(properties="french") as client:
```

--------------------------------

### Start CoreNLP Client with Properties File

Source: https://stanfordnlp.github.io/stanza/client_properties.html

Use a path to a properties file for the finest control over annotators and models. This method is recommended for complex configurations.

```python
with CoreNLPClient(properties='/path/to/server.props') as client:

    pass
```

--------------------------------

### Install CoreNLP Automatically

Source: https://stanfordnlp.github.io/stanza/client_setup

Use this function to automatically install CoreNLP and default models. CoreNLP is installed in `~/stanza_corenlp` by default.

```python
import stanza
stanza.install_corenlp()
```

--------------------------------

### Example of Finding Common Trees

Source: https://stanfordnlp.github.io/stanza/new_language_constituency.html

An example demonstrating the usage of the `common_trees` script to find common trees between two specific files, likely generated from different model configurations.

```bash
python3 -m stanza.utils.datasets.constituency.common_trees AA.topdown.mrg AA.inorder.mrg > AA.both.mrg

```

--------------------------------

### Tokenizer Training Example

Source: https://stanfordnlp.github.io/stanza/training_and_evaluation.html

Example command to train a tokenizer on the UD_English-EWT corpus with specific batch size and dropout settings.

```bash
python -m stanza.utils.training.run_tokenize UD_English-EWT --batch_size 32 --dropout 0.33
```

--------------------------------

### Install Stanza Dependencies

Source: https://stanfordnlp.github.io/stanza/new_language_coref.html

Install the necessary Python dependencies for Stanza using pip. This command assumes you are in the root of the cloned repository.

```bash
pip install .
```

--------------------------------

### Start Coreference Model Training

Source: https://stanfordnlp.github.io/stanza/new_language_coref.html

Initiates the training process for a coreference model. Replace `[your experiment]` with the name of the experiment defined in your TOML configuration file.

```bash
python -m stanza.models.wl_coref train [your experiment] --config_file ./data/coref_config.toml

```

--------------------------------

### Prepare Tokenizer Treebank and Run Tokenizer Training

Source: https://stanfordnlp.github.io/stanza/update_ud.html

Prepare treebanks specifically for the tokenizer and then run the tokenizer training script. This is an example for a combined English model.

```python
python3 stanza/utils/datasets/prepare_tokenizer_treebank.py en_combined
```

```python
python3 stanza/utils/training/run_tokenizer.py en_combined
```

--------------------------------

### Install Dependencies for Deduplication

Source: https://stanfordnlp.github.io/stanza/new_language_charlm.html

Install the required Python packages for running the Community-OSCAR deduplication script.

```bash
pip install datasketch huggingface_hub zstandard
```

--------------------------------

### CoreNLP Dependency Parse Structure Example

Source: https://stanfordnlp.github.io/stanza/client_usage.html

This is an example output format for a dependency parse, showing 'node' and 'edge' structures with details like sentence index, source, target, and dependency relation.

```protobuf
node {
  sentenceIndex: 0
  index: 1
}
...
edge {
  source: 2
  target: 1
  dep: "compound"
  isExtra: false
  sourceCopy: 0
  targetCopy: 0
  language: UniversalEnglish
}
...
root: 6

```

--------------------------------

### Coreference Chain Structure Example

Source: https://stanfordnlp.github.io/stanza/client_usage.html

Example output for a coreference chain, showing mention details like ID, type, and the representative mention for the cluster.

```protobuf
Chris Manning
[
mention {
  mentionID: 0
  mentionType: "PROPER"
  ...
}
mention {
  mentionID: 2
  mentionType: "PROPER"
  ...
}
mention {
  mentionID: 5
  mentionType: "PRONOMINAL"
  ...
}
representative: 0
...
]

```

--------------------------------

### Setup Stanza Test Data

Source: https://stanfordnlp.github.io/stanza/contributing.html

Run this script to download necessary models and a CoreNLP distribution for testing. Data is saved to 'stanza_test' by default, but can be configured with the TEST_HOME_VAR environment variable.

```python
python3 -m stanza.tests.setup

```

--------------------------------

### Basic Multilingual Pipeline Example

Source: https://stanfordnlp.github.io/stanza/langid.html

Process a list of texts with a MultilingualPipeline, which automatically detects the language and applies the appropriate language-specific Stanza pipeline. It caches pipelines for efficiency. This example demonstrates handling English and French texts.

```python
from stanza.pipeline.multilingual import MultilingualPipeline

nlp = MultilingualPipeline()
docs = ["Hello world!", "C'est une phrase française.", "This is an English sentence."]
docs = nlp(docs)
for doc in docs:
    print("---")
    print(f"text: {doc.text}")
    print(f"lang: {doc.lang}")
    print(f"{doc.sentences[0].dependencies_string()}")

```

--------------------------------

### CoreNLP Constituency Parse Structure Example

Source: https://stanfordnlp.github.io/stanza/client_usage.html

This is an example output format for a constituency parse tree, showing nested 'child' structures and 'value' fields.

```protobuf
child {
  child {
    child {
      child {
        value: "Chris"
      }
      value: "NNP"
      score: -9.281864166259766
    }
    ...
  }
  ...
  value: "S"
  score: -50.052059173583984
}
value: "ROOT"
score: -50.20326614379883

```

--------------------------------

### List CorefUD Data Files

Source: https://stanfordnlp.github.io/stanza/new_language_coref.html

Example of CorefUD data files (.conllu format) placed in the model-specific directory. Ensure train and dev files share the same base name.

```bash
$ ls -l ./extern_data/coref/my_coref
total 35328
-rw-r--r--  1 houjun  staff     73169 Mar 28 06:21 en_parcorfull-corefud-dev.conllu
-rw-r--r--  1 houjun  staff    574717 Mar 28 06:21 en_parcorfull-corefud-train.conllu
-rw-r--r--  1 houjun  staff   1940312 Mar 28 06:21 fr_democrat-corefud-dev.conllu
-rw-r--r--  1 houjun  staff  15491875 Mar 28 06:21 fr_democrat-corefud-train.conllu
```

--------------------------------

### Example Snapshot Output Format

Source: https://stanfordnlp.github.io/stanza/new_language_charlm.html

The output format for snapshots, which can be used as input for the deduplication script.

```text
bn:2024-38
bn:2024-33
bn:2024-30
...
```

--------------------------------

### Initialize pipeline with custom POS model and vectors

Source: https://stanfordnlp.github.io/stanza/word_vectors.html

Example of initializing a Stanza pipeline with a custom POS model and corresponding word vectors.

```python
pipe = stanza.Pipeline(lang="en", processors="tokenize,pos", pos_pretrain_path="new_en_wv.pt", pos_model_path="new_pos_model.pt")
```

--------------------------------

### Use Semgrex on Depparse Dependencies with Stanza

Source: https://stanfordnlp.github.io/stanza/client_regex.html

This example demonstrates using Semgrex to search dependencies produced by Stanza's depparse module. It requires downloading the 'en' model and initializing the Stanza pipeline.

```python
import stanza
import stanza.server.semgrex as semgrex

# stanza.download("en")
lp = stanza.Pipeline("en", processors="tokenize,pos,lemma,depparse")

doc = nlp("Banning opal removed all artifact decks from the meta.  I miss playing lantern.")
semgrex_results = semgrex.process_doc(doc,
                                      "{pos:NN}=object <obl {}=action",
                                      "{cpos:NOUN}=thing <obj {cpos:VERB}=action")
print(semgrex_results)

```

--------------------------------

### Initialize CoreNLP Client with French Models

Source: https://stanfordnlp.github.io/stanza/client_properties.html

Start a CoreNLP client with default French models by specifying 'french' in the properties argument. This ensures the client uses the correct language models.

```python
with CoreNLPClient(properties='french') as client:
```

--------------------------------

### Start CoreNLP Client with Python Dictionary

Source: https://stanfordnlp.github.io/stanza/client_properties.html

Provide a Python dictionary to override default models or specify custom annotator models. This offers more control than just a language name.

```python
with CoreNLPClient(properties={
      'annotators': 'tokenize,ssplit,pos',
      'pos.model': '/path/to/custom-model.ser.gz'
  }) as client:

    pass
```

--------------------------------

### Start CoreNLP Client with Language Name

Source: https://stanfordnlp.github.io/stanza/client_properties.html

Use a language name to quickly switch between languages, relying on default models for that language. This is a convenient way to specify basic pipeline properties.

```python
with CoreNLPClient(properties='french') as client:

    pass
```

--------------------------------

### Download Specific Processors and Initialize Pipeline Offline

Source: https://stanfordnlp.github.io/stanza/getting_started

Downloads 'tokenize' and 'pos' processors for Chinese, then initializes the pipeline without attempting further downloads. This is useful for offline usage.

```python
import stanza
stanza.download('zh', processors='tokenize,pos')
nlp = stanza.Pipeline('zh', processors='tokenize,pos', download_method=None)
```

--------------------------------

### Prepare Datasets and Run Training Scripts

Source: https://stanfordnlp.github.io/stanza/update_ud.html

Prepare datasets and run training scripts for annotators like tokenizer, MWT, POS, lemma, and depparse. The POS annotator must be trained before the depparse annotator due to tag dependencies.

```python
python3 stanza/utils/datasets/prepare_annotator_treebank.py ud_all
```

```python
python3 stanza/utils/training/run_annotator.py ud_all
```

--------------------------------

### Download and Organize Dataset

Source: https://stanfordnlp.github.io/stanza/new_language_ner.html

Download the dataset into the `$NERBASE` directory, organizing it under a language-specific subdirectory (e.g., 'bangla'). This step ensures data is stored in a structured and accessible manner.

```bash
cd $NERBASE
mkdir -p bangla
cd bangla
git clone git@github.com:Rifat1493/Bengali-NER.git

```

--------------------------------

### Install Stanza via Anaconda

Source: https://stanfordnlp.github.io/stanza/installation_usage

Install Stanza using the conda package manager. Note: This method may not be compatible with Python 3.8.

```bash
conda install -c stanfordnlp stanza

```

--------------------------------

### Download and Load English Pipeline with WARN Logging

Source: https://stanfordnlp.github.io/stanza/getting_started

Downloads the English pipeline and loads it, printing only warnings and errors during the process. Specify the logging level using the `logging_level` argument.

```python
stanza.download('en', logging_level='WARN')
nlp = stanza.Pipeline('en', logging_level='WARN')
```

--------------------------------

### Example UD dataset file naming convention

Source: https://stanfordnlp.github.io/stanza/training_and_evaluation.html

Stanza expects specific file naming patterns for train, dev, and test splits, as shown in this example for the EWT corpus.

```text
en_ewt-ud-train.conllu
en_ewt-ud-dev.conllu
en_ewt-ud-test.conllu
en_ewt-ud-train.txt
en_ewt-ud-dev.txt
en_ewt-ud-test.txt
```

--------------------------------

### Initialize Stanza Pipeline and Process Text

Source: https://stanfordnlp.github.io/stanza/index.html

This Python code demonstrates how to import Stanza, download a language model (English in this case), initialize a neural pipeline, and process a sample sentence.

```python
import stanza
stanza.download('en') # download English model
lp = stanza.Pipeline('en') # initialize English neural pipeline
doc = nlp("Barack Obama was born in Hawaii.") # run annotation over a sentence

```

--------------------------------

### Download Custom Packages and Initialize Pipeline Offline

Source: https://stanfordnlp.github.io/stanza/getting_started

Downloads custom-specified packages for German processors and initializes the pipeline without attempting downloads. This ensures all necessary models are available locally.

```python
processor_dict = {
    'tokenize': 'gsd', 
    'pos': 'hdt',
    'ner': 'conll03',
    'lemma': 'default'
}
stanza.download('de', processors=processor_dict, package=None)
nlp = stanza.Pipeline('de', processors=processor_dict, package=None, download_method=None)
```

--------------------------------

### Basic Stanza Pipeline Initialization and Annotation

Source: https://stanfordnlp.github.io/stanza/getting_started

Builds an English Stanza pipeline with tokenization and POS tagging processors, configured to use the GPU and limit POS tagging batch size. It then annotates a sample sentence and prints the result.

```python
import stanza

lp = stanza.Pipeline('en', processors='tokenize,pos', use_gpu=True, pos_batch_size=3000) # Build the pipeline, specify part-of-speech processor's batch size
doc = nlp("Barack Obama was born in Hawaii.") # Run the pipeline on the input text
print(doc) # Look at the result
```

--------------------------------

### Basic Language ID Example

Source: https://stanfordnlp.github.io/stanza/langid.html

Identify the language of text using the 'langid' processor. The detected language is stored in the 'lang' field of the Document object. This example processes a list of strings and prints their detected languages.

```python
from stanza.models.common.doc import Document
from stanza.pipeline.core import Pipeline

lp = Pipeline(lang="multilingual", processors="langid")
docs = ["Hello world.", "Bonjour le monde!"]
docs = [Document([], text=text) for text in docs]
nlp(docs)
print("\n".join(f"{doc.text}\t{doc.lang}" for doc in docs))

```

--------------------------------

### Download and Use CRAFT Pipeline

Source: https://stanfordnlp.github.io/stanza/biomed_model_usage

Initializes the CRAFT pipeline for syntactic analysis. Downloads the necessary package if not already present. Use this for general biomedical text annotation.

```python
import stanza
# download and initialize the CRAFT pipeline
stanza.download('en', package='craft')
nlp = stanza.Pipeline('en', package='craft')
# annotate example text
doc = nlp('A single-cell transcriptomic atlas characterizes ageing tissues in the mouse.')
# print out dependency tree
doc.sentences[0].print_dependencies()

```

--------------------------------

### Import CoreNLPClient

Source: https://stanfordnlp.github.io/stanza/client_properties.html

The first step to using the CoreNLP client is to import the necessary class.

```python
from stanza.server import CoreNLPClient
```

--------------------------------

### Upgrade Stanza via pip

Source: https://stanfordnlp.github.io/stanza/installation_usage

If you have a previous version of Stanza installed, use this command to upgrade to the latest version.

```bash
pip install stanza -U

```

--------------------------------

### Pretokenized Output Example

Source: https://stanfordnlp.github.io/stanza/tokenize.html

This output demonstrates that no further tokenization or sentence segmentation is performed when `tokenize_pretokenized` is enabled. Punctuation remains attached to tokens.

```text
====== Sentence 1 tokens =======
id: 1	text: This
id: 2	text: is
id: 3	text: token.ization
id: 4	text: done
id: 5	text: my
id: 6	text: way!
====== Sentence 2 tokens =======
id: 1	text: Sentence
id: 2	text: split,
id: 3	text: too!
```

--------------------------------

### Deploy models and run NER

Source: https://stanfordnlp.github.io/stanza/new_language_charlm.html

Copy the trained model files to the Stanza resources directory and execute the NER training script.

```bash
mkdir -p ~/stanza_resources/bn/forward_charlm
cp saved_models/charlm/bn_oscar_forward_charlm.pt ~/stanza_resources/bn/forward_charlm/oscar.pt

mkdir -p ~/stanza_resources/bn/backward_charlm
cp saved_models/charlm/bn_oscar_backward_charlm.pt ~/stanza_resources/bn/backward_charlm/oscar.pt

python3 -m stanza.utils.training.run_ner bn_daffodil --charlm oscar --save_name bn_daffodil_charlm.pt
```

--------------------------------

### Initialize Default English Pipeline

Source: https://stanfordnlp.github.io/stanza/getting_started

Initializes a Stanza pipeline with default processors for English. This is the most basic way to start annotating text.

```python
import stanza
nlp = stanza.Pipeline('en')
```

--------------------------------

### Launch Server with Custom Parser Model and JSON Output

Source: https://stanfordnlp.github.io/stanza/client_properties.html

Initialize a CoreNLP server with a specific parser model and set the output format to JSON. This is useful for custom parsing needs.

```python
CUSTOM_PROPS = {"parse.model": "edu/stanford/nlp/models/srparser/englishSR.beam.ser.gz"}

with CoreNLPClient(properties=CUSTOM_PROPS, output_format="json") as client:
```

--------------------------------

### Prepare Constituency Dataset Wrapper Script

Source: https://stanfordnlp.github.io/stanza/new_language_constituency.html

Command-line interface for preparing constituency datasets. This script calls dataset-specific preparation scripts, allowing for different languages to be processed.

```bash
python3 -m stanza.utils.datasets.constituency.prepare_con_dataset it_vit
python3 -m stanza.utils.datasets.constituency.prepare_con_dataset vi_vlsp22
etc etc

```

--------------------------------

### Example PTB Bracketed Tree

Source: https://stanfordnlp.github.io/stanza/new_language_constituency.html

Internal representation of data for the constituency parser. This format uses parentheses to denote subtrees and labels for each node.

```text
(ROOT (S (NP (DT This)) (VP (VBZ is) (NP (DT a) (NN sample)))))

```

--------------------------------

### Train Constituency Model

Source: https://stanfordnlp.github.io/stanza/new_language_constituency.html

Use this command to start training a constituency model for a specific language. Training can be time-consuming and is recommended to be performed on a GPU.

```python
python -m stanza.utils.training.run_constituency it_vit

```

--------------------------------

### Default Sentence Segmentation Behavior

Source: https://stanfordnlp.github.io/stanza/tokenize.html

This output contrasts with the `tokenize_no_ssplit=True` example, showing Stanza's default behavior where sentence segmentation is performed normally.

```text
====== Sentence 1 tokens =======
id: 1	text: This
id: 2	text: is
id: 3	text: a
id: 4	text: sentence
id: 5	text: .
====== Sentence 2 tokens =======
id: 1	text: This
id: 2	text: is
id: 3	text: a
id: 4	text: second
id: 5	text: .
====== Sentence 3 tokens =======
id: 1	text: This
id: 2	text: is
id: 3	text: a
id: 4	text: third
id: 5	text: .
```

--------------------------------

### Prepare CharLM dataset

Source: https://stanfordnlp.github.io/stanza/new_language_charlm.html

Converts raw text files in the specified directory into a dataset suitable for CharLM training.

```bash
python3 -m stanza.utils.charlm.make_lm_data extern_data/charlm_raw extern_data/charlm
```

--------------------------------

### Example NER Token Output

Source: https://stanfordnlp.github.io/stanza/ner.html

Illustrates the expected output format for NER tokens, including ID, text, character offsets, and NER tags.

```json
[
  {
    "id": 1,
    "text": "John",
    "start_char": 0,
    "end_char": 4,
    "ner": "B-PERSON",
    "multi_ner": [
      "O",
      "B-PERSON"
    ]
  }
], [
  {
    "id": 2,
    "text": "Bauer",
    "start_char": 5,
    "end_char": 10,
    "ner": "E-PERSON",
    "multi_ner": [
      "O",
      "E-PERSON"
    ]
  }
]]
```

```json
[
  {
    "id": 8,
    "text": "hip",
    "start_char": 37,
    "end_char": 40,
    "ner": "B-DISEASE",
    "multi_ner": [
      "B-DISEASE",
      "O"
    ]
  }
], [
  {
    "id": 9,
    "text": "arthritis",
    "start_char": 41,
    "end_char": 50,
    "ner": "E-DISEASE",
    "multi_ner": [
      "E-DISEASE",
      "O"
    ]
  }
]]
```

--------------------------------

### Initialize Pipeline with Resource Reuse

Source: https://stanfordnlp.github.io/stanza/getting_started

Initializes a Chinese pipeline, attempting to reuse existing resources before downloading. This method is efficient for environments with pre-existing Stanza models.

```python
from stanza.pipeline.core import DownloadMethod
nlp = stanza.Pipeline('zh', processors='tokenize,pos', download_method=DownloadMethod.REUSE_RESOURCES)
```

--------------------------------

### Start Basic NER Training

Source: https://stanfordnlp.github.io/stanza/new_language_ner.html

Initiates the training process for a Bengali NER model named 'bn_daffodil'. The trained model will be saved in the 'saved_models/ner/' directory.

```bash
python -m stanza.utils.training.run_ner bn_daffodil
```

--------------------------------

### Prepare and Combine Datasets

Source: https://stanfordnlp.github.io/stanza/retrain_ner.html

This command prepares both CoNLL03 and Worldwide datasets and then combines their training data into a single file.

```bash
python3 stanza/utils/datasets/ner/prepare_ner_dataset.py en_conll03ww

```

--------------------------------

### Accessing NER Tokens in Stanza

Source: https://stanfordnlp.github.io/stanza/ner.html

Demonstrates how to access specific tokens within a sentence for NER analysis. Note that token IDs start from 1.

```python
print(doc.sentences[0].tokens[0:2])
```

```python
print(doc.sentences[0].tokens[8:10])
```

--------------------------------

### Prepare Tokenizer Data

Source: https://stanfordnlp.github.io/stanza/retrain_ud.html

Use this script to read your dataset from `$UDBASE` and write it to `$TOKENIZE_DATA_DIR` for tokenizer training.

```python
python3 -m stanza.utils.datasets.prepare_tokenizer_treebank UD_English-EWT

```

--------------------------------

### Tokenization and Sentence Segmentation Example

Source: https://stanfordnlp.github.io/stanza/tokenize.html

Use this snippet to perform standard tokenization and sentence segmentation. The input text is split into sentences, and each sentence is further split into tokens.

```python
import stanza

lp = stanza.Pipeline(lang='en', processors='tokenize')
doc = nlp('This is a test sentence for stanza. This is another sentence.')
for i, sentence in enumerate(doc.sentences):
    print(f'====== Sentence {i+1} tokens =======')
    print(*[f'id: {token.id}\ttext: {token.text}' for token in sentence.tokens], sep='\n')
```

```text
====== Sentence 1 tokens =======
id: 1	text: This
id: 2	text: is
id: 3	text: a
id: 4	text: test
id: 5	text: sentence
id: 6	text: for
id: 7	text: stanza
id: 8	text: .
====== Sentence 2 tokens =======
id: 1	text: This
id: 2	text: is
id: 3	text: another
id: 4	text: sentence
id: 5	text: .
```

--------------------------------

### Run Constituency Dataset Preparation

Source: https://stanfordnlp.github.io/stanza/training_and_evaluation.html

Executes the constituency dataset preparation script, optionally specifying a dataset like ja_alt.

```bash
python3 -m stanza.utils.datasets.constituency.prepare_con_dataset
python3 -m stanza.utils.datasets.constituency.prepare_con_dataset ja_alt
```

--------------------------------

### Run Semgrex Patterns with CoreNLP Client

Source: https://stanfordnlp.github.io/stanza/client_regex.html

Utilize Semgrex patterns to directly extract relationships between words, such as subject-object pairs. The client setup is similar to TokensRegex.

```python
text = "Chris Manning is a nice person. Chris wrote a simple sentence. He also gives oranges to people."
with CoreNLPClient(
        annotators=['tokenize','ssplit','pos','lemma','ner', 'parse', 'depparse'],
        timeout=30000,
        memory='16G') as client:

    # Use semgrex patterns to directly find who wrote what.
    pattern = '{word:wrote} >nsubj {}=subject >obj {}=object'
    matches = client.semgrex(text, pattern)
    # sentences contains a list with matches for each sentence.
    print(len(matches["sentences"]))
    # length tells you whether or not there are any matches in this
    print(matches["sentences"][1]["length"])
    # You can access matches like most regex groups.
    print(matches["sentences"][1]["0"]["text"])
    print(matches["sentences"][1]["0"]["$subject"]["text"])
    print(matches["sentences"][1]["0"]["$object"]["text"])

```

--------------------------------

### Prepare SST Sentiment Dataset

Source: https://stanfordnlp.github.io/stanza/retrain_sentiment.html

Use the prepare_sentiment_dataset.py script to convert the SST dataset into a JSON format suitable for the Stanza classifier. This script requires CoreNLP to be installed.

```python
python3 stanza/utils/datasets/sentiment/prepare_sentiment_dataset.py en_sst3

```

--------------------------------

### Initialize CoreNLPClient with Custom Options

Source: https://stanfordnlp.github.io/stanza/client_properties.html

Use this snippet to initialize the CoreNLPClient with specific annotators, memory allocation, output format, and logging preferences. The `be_quiet=False` option is useful for debugging server-side errors.

```python
with CoreNLPClient(
    annotators='tokenize,ssplit,pos,lemma,ner',
    output_format='text',
    memory='8G',
    be_quiet=False) as client:


```

--------------------------------

### Initialize Stanza Pipeline with Constituency Processor

Source: https://stanfordnlp.github.io/stanza/constituency.html

Use this to initialize the Stanza pipeline with the constituency processor enabled. Ensure 'tokenize' and 'pos' processors are included as they are requirements.

```python
import stanza

nlp = stanza.Pipeline(lang='en', processors='tokenize,pos,constituency')
doc = nlp('This is a test')
for sentence in doc.sentences:
    print(sentence.constituency)

```

--------------------------------

### Checking for No Semgrex Matches

Source: https://stanfordnlp.github.io/stanza/client_regex.html

If a sentence has no matches for a given semgrex query, the result list will be empty. This example demonstrates checking the length of the match list for an empty result.

```python
print(len(semgrex_results.result[1].result[0].match))
```

```text
0
```

--------------------------------

### Download and Use MIMIC Pipeline with i2b2 NER

Source: https://stanfordnlp.github.io/stanza/biomed_model_usage

Initializes a pipeline with MIMIC syntactic analysis models and the i2b2 clinical NER model. Downloads required packages. Use this for clinical text analysis and entity extraction.

```python
# download and initialize a mimic pipeline with an i2b2 NER model
stanza.download('en', package='mimic', processors={'ner': 'i2b2'})
nlp = stanza.Pipeline('en', package='mimic', processors={'ner': 'i2b2'})
# annotate clinical text
doc = nlp('The patient had a sore throat and was treated with Cepacol lozenges.')
# print out all entities
for ent in doc.entities:
    print(f'{ent.text}\t{ent.type}')

```

--------------------------------

### Configure Training Data Paths

Source: https://stanfordnlp.github.io/stanza/new_language_coref.html

Sets the paths for the training, development, and testing datasets within the coreference configuration file. Ensure these files exist.

```toml
train_data = "data/coref/my_coref.train.json"
dev_data = "data/coref/my_coref.dev.json"
test_data = "data/coref/my_coref.dev.json"

```

--------------------------------

### Download CoreNLP Models for a Specific Language

Source: https://stanfordnlp.github.io/stanza/client_setup

Downloads additional CoreNLP models for specified languages and versions. The `dir` argument is required if using a customized CoreNLP installation location.

```python
stanza.download_corenlp_models(model='french', version='4.2.2', dir="YOUR_CORENLP_FOLDER")
```

--------------------------------

### Push Stanza Models to HuggingFace

Source: https://stanfordnlp.github.io/stanza/update_ud.html

Navigate to the software directory and execute the Python script to push all models to HuggingFace. Ensure you are in the correct directory before running the script.

```bash
cd /u/nlp/software/
python3 huggingface-models/hugging_stanza.py

```

--------------------------------

### Run Sentiment Training

Source: https://stanfordnlp.github.io/stanza/new_language_sentiment.html

Initiate the sentiment model training process. The dataset name should be provided as an argument. The script will attempt to automatically download necessary models like Spanish word vectors or charlm.

```bash
python -m stanza.utils.training.run_sentiment es_tass2020
```

--------------------------------

### Create a New Dataset Conversion Script

Source: https://stanfordnlp.github.io/stanza/new_language_coref.html

Guidance on creating a custom conversion script for unsupported datasets. The script should transform data into .json files compatible with Stanza's coreference models, similar to `convert_hindi.py`.

```python
# Example structure for a new dataset conversion script
# You will need to create a Pipeline for the language in question.
# Once that is created, you will want to split each document into lists of words for each sentence, 
# lists of coref chains for each sentence (see the format used in the process_document utility function), 
# and lists of speakers (which can be "" if no speakers are annotated).
# The utility function process_document will use the pipeline to find the heads of the coref phrases 
# and turn the document into a json-ready document. You can then write this out with the python json library, 
# as done in `convert_hindi.py`.

# One setting you may want to experiment with is the usage of `cconj`.
# Being the conjunction-aware word level coref tool, this model uses conjunctions to mark the heads of phrases.
# However, while that has worked well in the languages we’ve experimented with, it won’t necessarily help with all languages.
```
Tokens:
Raw