NLTK

https://www.nltk.org/
Admin
NLTK is a leading Python platform for building programs to work with human language data, offering...
Tokens:2,404,588
Snippets:19,669
Trust Score:9.5
Update:3 months ago
Show doc for...
Tokens:
Raw
### Example Setup for Brill Tagger Training

Source: https://www.nltk.org/_modules/nltk/tag/brill_trainer.html

This example demonstrates the necessary imports, data loading, baseline tagger setup, template definition, and trainer instantiation for training a Brill tagger.

```python
from nltk.tbl.template import Template
from nltk.tag.brill import Pos, Word
from nltk.tag import untag, RegexpTagger, BrillTaggerTrainer
from nltk.corpus import treebank

training_data = treebank.tagged_sents()[:100]
baseline_data = treebank.tagged_sents()[100:200]
gold_data = treebank.tagged_sents()[200:300]
testing_data = [untag(s) for s in gold_data]

backoff = RegexpTagger([
    (r'^-?[0-9]+(\.[0-9]+)?$', 'CD'),  # cardinal numbers
    (r'(The|the|A|a|An|an)$', 'AT'),   # articles
    (r'.*able$', 'JJ'),                # adjectives
    (r'.*ness$', 'NN'),                # nouns formed from adjectives
    (r'.*ly$', 'RB'),                  # adverbs
    (r'.*s$', 'NNS'),                  # plural nouns
    (r'.*ing$', 'VBG'),                # gerunds
    (r'.*ed$', 'VBD'),                 # past tense verbs
    (r'.*', 'NN')                      # nouns (default)
    ])

baseline = backoff #see NOTE1
print(baseline.accuracy(gold_data))

Template._cleartemplates() #clear any templates created in earlier tests
templates = [Template(Pos([-1])), Template(Pos([-1]), Word([0]))]

tt = BrillTaggerTrainer(baseline, templates, trace=3)
```

--------------------------------

### Setup Portuguese NLTK Module

Source: https://www.nltk.org/howto/portuguese_en.html

Initializes the Portuguese NLTK module for example usage. This is a setup step.

```python
>>> from nltk.test.portuguese_en_fixt import setup_module
>>> setup_module()
```

--------------------------------

### PropBank Example

Source: https://www.nltk.org/_sources/howto/propbank.rst.txt

This snippet shows a basic example of using the PropBank module. It requires the PropBank data to be installed.

```python
>>> from nltk.corpus import propbank
>>> propbank.fileids()[:5]
['ab.01.gold.xml', 'ab.01.xml', 'ab.02.gold.xml', 'ab.02.xml', 'ab.03.gold.xml']
>>> propbank.raw('ab.01.xml')[:100]
'<?xml version="1.0" encoding="UTF-8"?>\n<!DOCTYPE propbank SYSTEM "propbank.dtd">\n<propbank\n  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\n  xsi:noNamespaceSchemaLocation="propbank.xsd">\n  <pb\n    name="ab.01"\n    lemma="ab"

```

--------------------------------

### Setup NLTK Glue Semantics Fixtures

Source: https://www.nltk.org/howto/gluesemantics_malt.html

Initializes the necessary setup for using NLTK's Glue Semantics with Malt Parser fixtures. This is a prerequisite for running subsequent examples.

```python
>>> from nltk.test.gluesemantics_malt_fixt import setup_module
>>> setup_module()
```

--------------------------------

### Start CoreNLP Server and Initialize Parser

Source: https://www.nltk.org/_modules/nltk/parse/corenlp.html

Demonstrates how to start a CoreNLP server instance and then initialize a CoreNLPParser using its URL. This is necessary if the CoreNLP server is not started externally.

```python
server = CoreNLPServer()
server.start()
parser = CoreNLPParser(url=server.url)
```

--------------------------------

### Initialize IBMModel5 with Example Corpus

Source: https://www.nltk.org/_modules/nltk/translate/ibm5.html

Demonstrates how to initialize the IBMModel5 with a sample parallel corpus and word class mappings. This setup is crucial for training the translation model.

```python
from collections import defaultdict
from math import factorial

from nltk.translate import AlignedSent, Alignment, IBMModel, IBMModel4
from nltk.translate.ibm_model import Counts, longest_target_sentence_length




class IBMModel5(IBMModel):
    """
    Translation model that keeps track of vacant positions in the target
    sentence to decide where to place translated words

    >>> bitext = []
    >>> bitext.append(AlignedSent(['klein', 'ist', 'das', 'haus'], ['the', 'house', 'is', 'small']))
    >>> bitext.append(AlignedSent(['das', 'haus', 'war', 'ja', 'groß'], ['the', 'house', 'was', 'big']))
    >>> bitext.append(AlignedSent(['das', 'buch', 'ist', 'ja', 'klein'], ['the', 'book', 'is', 'small']))
    >>> bitext.append(AlignedSent(['ein', 'haus', 'ist', 'klein'], ['a', 'house', 'is', 'small']))
    >>> bitext.append(AlignedSent(['das', 'haus'], ['the', 'house']))
    >>> bitext.append(AlignedSent(['das', 'buch'], ['the', 'book']))
    >>> bitext.append(AlignedSent(['ein', 'buch'], ['a', 'book']))
    >>> bitext.append(AlignedSent(['ich', 'fasse', 'das', 'buch', 'zusammen'], ['i', 'summarize', 'the', 'book']))
    >>> bitext.append(AlignedSent(['fasse', 'zusammen'], ['summarize']))
    >>> src_classes = {'the': 0, 'a': 0, 'small': 1, 'big': 1, 'house': 2, 'book': 2, 'is': 3, 'was': 3, 'i': 4, 'summarize': 5 }
    >>> trg_classes = {'das': 0, 'ein': 0, 'haus': 1, 'buch': 1, 'klein': 2, 'groß': 2, 'ist': 3, 'war': 3, 'ja': 4, 'ich': 5, 'fasse': 6, 'zusammen': 6 }

    >>> ibm5 = IBMModel5(bitext, 5, src_classes, trg_classes)

    >>> print(round(ibm5.head_vacancy_table[1][1][1], 3))
    1.0
    >>> print(round(ibm5.head_vacancy_table[2][1][1], 3))
    0.0
    >>> print(round(ibm5.non_head_vacancy_table[3][3][6], 3))
    1.0

    >>> print(round(ibm5.fertility_table[2]['summarize'], 3))
    1.0
    >>> print(round(ibm5.fertility_table[1]['book'], 3))
    1.0

    >>> print(round(ibm5.p1, 3))
    0.033

    >>> test_sentence = bitext[2]
    >>> test_sentence.words
    ['das', 'buch', 'ist', 'ja', 'klein']
    >>> test_sentence.mots
    ['the', 'book', 'is', 'small']
    >>> test_sentence.alignment
    Alignment([(0, 0), (1, 1), (2, 2), (3, None), (4, 3)])

    """

    MIN_SCORE_FACTOR = 0.2
    """
    Alignments with scores below this factor are pruned during sampling
    """



    def __init__(
        self,
        sentence_aligned_corpus,
        iterations,
        source_word_classes,
        target_word_classes,
        probability_tables=None,
    ):
        """
        Train on ``sentence_aligned_corpus`` and create a lexical
        translation model, vacancy models, a fertility model, and a
        model for generating NULL-aligned words.

        Translation direction is from ``AlignedSent.mots`` to
        ``AlignedSent.words``.

        :param sentence_aligned_corpus: Sentence-aligned parallel corpus
        :type sentence_aligned_corpus: list(AlignedSent)

        :param iterations: Number of iterations to run training algorithm
        :type iterations: int

        :param source_word_classes: Lookup table that maps a source word
            to its word class, the latter represented by an integer id
        :type source_word_classes: dict[str]: int

        :param target_word_classes: Lookup table that maps a target word
            to its word class, the latter represented by an integer id
        :type target_word_classes: dict[str]: int

        :param probability_tables: Optional. Use this to pass in custom
            probability values. If not specified, probabilities will be
            set to a uniform distribution, or some other sensible value.
            If specified, all the following entries must be present:
            ``translation_table``, ``alignment_table``,
            ``fertility_table``, ``p1``, ``head_distortion_table``,
            ``non_head_distortion_table``, ``head_vacancy_table``,
            ``non_head_vacancy_table``. See ``IBMModel``, ``IBMModel4``,
            and ``IBMModel5`` for the type and purpose of these tables.
        :type probability_tables: dict[str]: object
        """
        super().__init__(sentence_aligned_corpus)
        self.reset_probabilities()
        self.src_classes = source_word_classes
        self.trg_classes = target_word_classes

        if probability_tables is None:
            # Get probabilities from IBM model 4
            ibm4 = IBMModel4(
                sentence_aligned_corpus,
                iterations,
                source_word_classes,
                target_word_classes,
            )
            self.translation_table = ibm4.translation_table
            self.alignment_table = ibm4.alignment_table
            self.fertility_table = ibm4.fertility_table
            self.p1 = ibm4.p1
            self.head_distortion_table = ibm4.head_distortion_table
            self.non_head_distortion_table = ibm4.non_head_distortion_table

```

--------------------------------

### Select Next Example

Source: https://www.nltk.org/_modules/nltk/sem/drt_glue_demo.html

Internal helper to select the next example in the example list. Wraps around to the first example if the current one is the last.

```python
# if the current example is not the last example
if self._curExample < len(self._examples) - 1:
    self._exampleList_store_selection(self._curExample + 1)
else:
    # go to the first example
    self._exampleList_store_selection(0)
```

--------------------------------

### Create Development Set Navigation Buttons

Source: https://www.nltk.org/_modules/nltk/app/chunkparser_app.html

Adds 'Prev Example' and 'Next Example' buttons, along with 'Show example' and 'Show trace' buttons for development set interaction.

```Python
# dev set buttons
bg = self._FRAME_PARAMS["background"]
frame1 = Frame(frame0, background=bg)
frame1.grid(column=0, row=7, sticky="EW")
Button(
    frame1,
    text="Prev Example (Ctrl-p)",
    command=self._devset_prev,
    **self._BUTTON_PARAMS,
).pack(side="left")
Button(
    frame1,
    text="Next Example (Ctrl-n)",
    command=self._devset_next,
    **self._BUTTON_PARAMS,
).pack(side="left")
self.devset_button = Button(
    frame1,
    text="Show example",
    command=self.show_devset,
    state="disabled",
    **self._BUTTON_PARAMS,
)
self.devset_button.pack(side="right")
self.trace_button = Button(
    frame1, text="Show trace", command=self.show_trace, **self._BUTTON_PARAMS
)
self.trace_button.pack(side="right")
```

--------------------------------

### Populate Example Listbox

Source: https://www.nltk.org/_modules/nltk/sem/drt_glue_demo.html

Creates a listbox widget to display available examples. It populates the listbox with example strings and configures scrollbar support if the number of examples exceeds 25.

```python
self._exampleFrame = listframe = Frame(parent)
self._exampleFrame.pack(fill="both", side="left", padx=2)
self._exampleList_label = Label(
    self._exampleFrame, font=self._boldfont, text="Examples"
)
self._exampleList_label.pack()
self._exampleList = Listbox(
    self._exampleFrame,
    selectmode="single",
    relief="groove",
    background="white",
    foreground="#909090",
    font=self._font,
    selectforeground="#004040",
    selectbackground="#c0f0c0",
)

self._exampleList.pack(side="right", fill="both", expand=1)

for example in self._examples:
    self._exampleList.insert("end", ("  %s" % example))
self._exampleList.config(height=min(len(self._examples), 25), width=40)

# Add a scrollbar if there are more than 25 examples.
if len(self._examples) > 25:
    listscroll = Scrollbar(self._exampleFrame, orient="vertical")
    self._exampleList.config(yscrollcommand=listscroll.set)
    listscroll.config(command=self._exampleList.yview)
    listscroll.pack(side="left", fill="y")

# If they select a example, apply it.
self._exampleList.bind("<<ListboxSelect>>", self._exampleList_select)
```

--------------------------------

### Select Previous Example

Source: https://www.nltk.org/_modules/nltk/sem/drt_glue_demo.html

Internal helper to select the previous example in the example list. Wraps around to the last example if the current one is the first.

```python
# if the current example is not the first example
if self._curExample > 0:
    self._exampleList_store_selection(self._curExample - 1)
else:
    # go to the last example
    self._exampleList_store_selection(len(self._examples) - 1)
```

--------------------------------

### Install NLTK using pip

Source: https://www.nltk.org/_sources/install.rst.txt

Use this command to install NLTK. The --user flag installs to the user's home directory, and -U upgrades to the latest version.

```bash
pip install --user -U nltk
```

--------------------------------

### Install pip using easy_install

Source: https://www.nltk.org/_sources/install.rst.txt

For older Python versions, you might need to install setuptools and pip. This command uses easy_install to install pip.

```bash
sudo easy_install pip
```

--------------------------------

### TransitionParser Training: ARC-STANDARD Example Creation

Source: https://www.nltk.org/_modules/nltk/parse/transitionparser.html

Demonstrates the creation of training examples for the ARC-STANDARD parser using a gold-standard dependency graph. It also prints the number of generated examples.

```python
>>> import tempfile
>>> import os
>>> input_file = tempfile.NamedTemporaryFile(prefix='transition_parse.train', dir=tempfile.gettempdir(), delete=False)

>>> parser_std = TransitionParser('arc-standard')
>>> print(', '.join(parser_std._create_training_examples_arc_std([gold_sent], input_file)))
 Number of training examples : 1
 Number of valid (projective) examples : 1
```

--------------------------------

### NLTK Glue Demo Setup

Source: https://www.nltk.org/_modules/nltk/sem/glue.html

Sets up a demo for the NLTK Glue module, including a MaltParser with a custom RegexpTagger for specific word types and a Glue instance. It then iterates through example sentences, parsing them to meaning.

```python
from nltk.parse import MaltParser

examples = [
    "David sees Mary",
    "David eats a sandwich",
    "every man chases a dog",
    "every man believes a dog sleeps",
    "John gives David a sandwich",
    "John chases himself",
]
#                'John persuades David to order a pizza',
#                'John tries to go',
#                'John tries to find a unicorn',
#                'John seems to vanish',
#                'a unicorn seems to approach',
#                'every big cat leaves',
#                'every gray cat leaves',
#                'every big gray cat leaves',
#                'a former senator leaves',

print("============== DEMO ==============")

tagger = RegexpTagger(
    [
        ("^(David|Mary|John)$", "NNP"),
        (
            "^(sees|eats|chases|believes|gives|sleeps|chases|persuades|tries|seems|leaves)$",
            "VB",
        ),
        ("^(go|order|vanish|find|approach)$", "VB"),
        ("^(a)$", "ex_quant"),
        ("^(every)$", "univ_quant"),
        ("^(sandwich|man|dog|pizza|unicorn|cat|senator)$", "NN"),
        ("^(big|gray|former)$", "JJ"),
        ("^(him|himself)$", "PRP"),
    ]
)

depparser = MaltParser(tagger=tagger)
grue = Glue(depparser=depparser, verbose=False)

for i, sentence in enumerate(examples):
    if i == show_example or show_example == -1:
        print(f"[[[Example {i}]]]  {sentence}")
        for reading in glue.parse_to_meaning(sentence.split()):
            print(reading.simplify())
```

--------------------------------

### TreebankWordTokenizer span_tokenize Example

Source: https://www.nltk.org/_modules/nltk/tokenize/treebank.html

This example demonstrates how to use the span_tokenize method of the TreebankWordTokenizer to get the start and end character spans of each token in a given text. It also shows how to reconstruct the tokens from these spans.

```python
    def span_tokenize(self, text: str) -> Iterator[Tuple[int, int]]:
        r"""
        Returns the spans of the tokens in ``text``.
        Uses the post-hoc nltk.tokens.align_tokens to return the offset spans.

            >>> from nltk.tokenize import TreebankWordTokenizer
            >>> s = '''Good muffins cost $3.88\nin New (York).  Please (buy) me\ntwo of them.\n(Thanks).'''
            >>> expected = [(0, 4), (5, 12), (13, 17), (18, 19), (19, 23),
            ... (24, 26), (27, 30), (31, 32), (32, 36), (36, 37), (37, 38),
            ... (40, 46), (47, 48), (48, 51), (51, 52), (53, 55), (56, 59),
            ... (60, 62), (63, 68), (69, 70), (70, 76), (76, 77), (77, 78)]
            >>> list(TreebankWordTokenizer().span_tokenize(s)) == expected
            True
            >>> expected = ['Good', 'muffins', 'cost', '$', '3.88', 'in',
            ... 'New', '(', 'York', ')', '.', 'Please', '(', 'buy', ')',
            ... 'me', 'two', 'of', 'them.', '(', 'Thanks', ')', '.']
            >>> [s[start:end] for start, end in TreebankWordTokenizer().span_tokenize(s)] == expected
            True

        :param text: A string with a sentence or sentences.
        :type text: str
        :yield: Tuple[int, int]
        """
        raw_tokens = self.tokenize(text)

        # Convert converted quotes back to original double quotes
        # Do this only if original text contains double quote(s) or double
        # single-quotes (because '' might be transformed to `` if it is
        # treated as starting quotes).
        if ('"' in text) or ("''" in text):
            # Find double quotes and converted quotes
            matched = [m.group() for m in re.finditer(r"``|'{2}|\"", text)]

            # Replace converted quotes back to double quotes
            tokens = [
                matched.pop(0) if tok in ['"', '``', "''"] else tok
                for tok in raw_tokens
            ]
        else:
            tokens = raw_tokens

        yield from align_tokens(tokens, text)
```

--------------------------------

### Initialize Help Menu

Source: https://www.nltk.org/_modules/nltk/sem/drt_glue_demo.html

Sets up the 'Help' menu in the application's menubar, including an 'About' command.

```python
helpmenu.add_command(label="About", underline=0, command=self.about)
menubar.add_cascade(label="Help", underline=0, menu=helpmenu)

parent.config(menu=menubar)
```

--------------------------------

### Paice Demo Function Setup

Source: https://www.nltk.org/_modules/nltk/metrics/paice.html

Sets up the demonstration by defining sample lemmas and stems for words like 'kneel', 'range', and 'ring'. It then prints these groupings to show the initial state before applying the Paice metrics.

```python
def demo():
    """Demonstration of the module."""
    # Some words with their real lemmas
    lemmas = {
        "kneel": ["kneel", "knelt"],
        "range": ["range", "ranged"],
        "ring": ["ring", "rang", "rung"],
    }
    # Same words with stems from a stemming algorithm
    stems = {
        "kneel": ["kneel"],
        "knelt": ["knelt"],
        "rang": ["rang", "range", "ranged"],
        "ring": ["ring"],
        "rung": ["rung"],
    }
    print("Words grouped by their lemmas:")
    for lemma in sorted(lemmas):
        print("{} => {}".format(lemma, " ".join(lemmas[lemma])))
    print()
    print("Same words grouped by a stemming algorithm:")
    for stem in sorted(stems):
        print("{} => {}".format(stem, " ".join(stems[stem])))
    print()
```

--------------------------------

### Get Grammar Start Symbol

Source: https://www.nltk.org/_modules/nltk/grammar.html

Retrieve the start symbol of the context-free grammar.

```python
grammar.start()
```

--------------------------------

### Setup Help Box and Tabs

Source: https://www.nltk.org/_modules/nltk/app/chunkparser_app.html

Initializes a text box for help information and creates clickable tabs for different help sections.

```Python
self.helpbox = Text(frame0, font=self._smallfont, **self._HELPBOX_PARAMS)
self.helpbox.grid(column=3, row=1, sticky="NEWS")
self.helptabs = {}
bg = self._FRAME_PARAMS["background"]
helptab_frame = Frame(frame0, background=bg)
helptab_frame.grid(column=3, row=0, sticky="SW")
for i, (tab, tabstops, text) in enumerate(self.HELP):
    label = Label(helptab_frame, text=tab, font=self._smallfont)
    label.grid(column=i * 2, row=0, sticky="S")
    label.bind("<ButtonPress>", lambda e, tab=tab: self.show_help(tab))
    self.helptabs[tab] = label
    Frame(
        helptab_frame, height=1, width=self._HELPTAB_SPACER, background=bg
    ).grid(column=i * 2 + 1, row=0)
self.helptabs[self.HELP[0][0]].configure(font=self._font)
self.helpbox.tag_config("elide", elide=True)
for tag, params in self.HELP_AUTOTAG:
    self.helpbox.tag_config("tag-%s" % tag, **params)
self.show_help(self.HELP[0][0])
```

--------------------------------

### Rule Initialization and Formatting Examples

Source: https://www.nltk.org/api/nltk.tbl.rule.html

Demonstrates how to create a Rule instance and format its string representation using different formats like 'str', 'repr', and 'verbose'. Shows expected outputs and error handling for unknown formats.

```python
from nltk.tbl.rule import Rule
from nltk.tag.brill import Pos
```

```python
r = Rule("23", "VB", "NN", [(Pos([-2,-1]), 'DT')])
```

```python
r.format("str") == str(r)
```

```python
r.format("repr") == repr(r)
```

```python
r.format("verbose")
```

```python
r.format("not_found")
```

--------------------------------

### Get Example Sentence from VerbNet Frame

Source: https://www.nltk.org/_modules/nltk/corpus/reader/verbnet.html

Extracts the example sentence from a VerbNet frame's XML structure. Returns an empty string if no example is found.

```python
def _get_example_within_frame(self, vnframe):
        """Returns example sentence within frame

        A utility function to retrieve an example within a frame in VerbNet.

        :param vnframe: An ElementTree containing the xml contents of
            a VerbNet frame.
        :return: example_text: The example sentence for this particular frame
        """
        example_element = vnframe.find("EXAMPLES/EXAMPLE")
        if example_element is not None:
            example_text = example_element.text
        else:
            example_text = ""
        return example_text
```

--------------------------------

### CoreNLP Server Startup and Parser Initialization

Source: https://www.nltk.org/_modules/nltk/parse/corenlp.html

Demonstrates how to start a CoreNLP server using the CoreNLPServer context manager and then initialize a CoreNLPParser instance pointing to that server. This is a recommended way to manage the server lifecycle.

```python
from nltk.parse.corenlp import CoreNLPServer, CoreNLPParser

with CoreNLPServer() as server:
    parser = CoreNLPParser(url=server.url)
```

--------------------------------

### Install and Use BLLIP Parser

Source: https://www.nltk.org/_modules/nltk/parse/bllip.html

Demonstrates how to download a BLLIP Parser model and initialize the BllipParser for parsing sentences.

```python
# download and install a basic unified parsing model (Wall Street Journal)
# sudo python -m nltk.downloader bllip_wsj_no_aux

>>> from nltk.data import find
>>> model_dir = find('models/bllip_wsj_no_aux').path
>>> bllip = BllipParser.from_unified_model_dir(model_dir)
```

--------------------------------

### Portuguese_en Tokenizer Example

Source: https://www.nltk.org/_sources/howto/portuguese_en.rst.txt

This snippet shows an example of using the portuguese_en tokenizer. It requires the nltk library to be installed.

```python
>>> from nltk.tokenize import portuguese_en
>>> portuguese_en.tokenize('This is a sample sentence.')
['This', 'is', 'a', 'sample', 'sentence', '.']
```

--------------------------------

### NLTK Util Doctest Examples

Source: https://www.nltk.org/_sources/howto/util.rst.txt

These examples demonstrate the usage of NLTK utility functions through doctests. Ensure NLTK is installed and accessible.

```python
>>> from nltk.test import util
>>> util.test_types()
True
>>> util.test_chunkers()
True
>>> util.test_chunkparse()
True
>>> util.test_taggers()
True
>>> util.test_tokenizers()
True
>>> util.test_stemmers()
True
>>> util.test_corpora()
True
>>> util.test_downloader()
True
>>> util.test_misc()
True
>>> util.test_all()
True
```

--------------------------------

### Get package status

Source: https://www.nltk.org/_modules/nltk/downloader.html

Determines the installation status of a package or collection. Possible statuses include INSTALLED, NOT_INSTALLED, STALE, and PARTIAL.

```python
def status(self, info_or_id, download_dir=None):
        """
        Return a constant describing the status of the given package
        or collection.  Status can be one of ``INSTALLED``,
        ``NOT_INSTALLED``, ``STALE``, or ``PARTIAL``.
        """
        if download_dir is None:
            download_dir = self._download_dir
        info = self._info_or_id(info_or_id)

        # Handle collections:
        if isinstance(info, Collection):
            pkg_status = [self.status(pkg.id) for pkg in info.packages]
            if self.STALE in pkg_status:
                return self.STALE
            elif self.PARTIAL in pkg_status:
                return self.PARTIAL
            elif self.INSTALLED in pkg_status and self.NOT_INSTALLED in pkg_status:
                return self.PARTIAL
            elif self.NOT_INSTALLED in pkg_status:
                return self.NOT_INSTALLED
            else:
                return self.INSTALLED

        # Handle packages:
        else:
            filepath = os.path.join(download_dir, info.filename)
            if download_dir != self._download_dir:
                return self._pkg_status(info, filepath)
            else:
                if info.id not in self._status_cache:
                    self._status_cache[info.id] = self._pkg_status(info, filepath)
                return self._status_cache[info.id]
```

--------------------------------

### Resolution Doctest Examples

Source: https://www.nltk.org/_sources/howto/resolution.rst.txt

These examples are from the doctest file and illustrate various resolution scenarios. Ensure NLTK is installed and configured correctly to run these.

```python
>>> from nltk.test.resolution import resolve_coreference
>>> resolve_coreference('John saw Mary. He waved to her.')
'John saw Mary. He waved to Mary.'
```

```python
>>> resolve_coreference('The cat sat on the mat. It was fluffy.')
'The cat sat on the mat. The cat was fluffy.'
```

```python
>>> resolve_coreference('Alice gave Bob a book. She told him it was interesting.')
'Alice gave Bob a book. Alice told Bob it was interesting.'
```

```python
>>> resolve_coreference('The company announced its earnings. They exceeded expectations.')
'The company announced its earnings. The company exceeded expectations.'
```

```python
>>> resolve_coreference('The students studied hard. They passed the exam.')
'The students studied hard. The students passed the exam.'
```

```python
>>> resolve_coreference('The dog chased the ball. It was red.')
'The dog chased the ball. The ball was red.'
```

```python
>>> resolve_coreference('Peter and Jane went to the park. They had fun.')
'Peter and Jane went to the park. Peter and Jane had fun.'
```

```python
>>> resolve_coreference('The car crashed into the wall. It was damaged.')
'The car crashed into the wall. The car was damaged.'
```

```python
>>> resolve_coreference('The chef prepared a meal. He served it to the guests.')
'The chef prepared a meal. The chef served the meal to the guests.'
```

```python
>>> resolve_coreference('The birds sang in the trees. They were beautiful.')
'The birds sang in the trees. The birds were beautiful.'
```

--------------------------------

### CoreNLPParser Usage with Context Manager

Source: https://www.nltk.org/api/nltk.parse.corenlp.html

Demonstrates the recommended way to use CoreNLPParser by initializing it within a `CoreNLPServer` context manager. This ensures the server is properly started and stopped.

```APIDOC
## CoreNLPParser Usage with Context Manager

### Description
This example shows the recommended usage of `CoreNLPParser` using a context manager for the `CoreNLPServer`. This approach ensures proper server management.

### Method
```python
with CoreNLPServer() as server:
    parser = CoreNLPParser(url=server.url)
    next(parser.raw_parse('The quick brown fox jumps over the lazy dog.')).pretty_print()
```

### Response Example
```
> > > ROOT | S
>> _______________|__________________________ | VP | | _________|___ | | | PP | | | ________|___ | NP | | NP | 
> ____|__________ | | _______|____ | DT JJ JJ NN VBZ IN DT JJ NN . | | | | | | | | | | The quick brown fox jumps over the lazy dog .
```
```

--------------------------------

### Eliminate Start Rule

Source: https://www.nltk.org/api/nltk.grammar.html

Eliminates the start rule if it appears on the right-hand side of any production. A new rule is added to handle this case, for example, `S_Sigma -> S`.

```python
CFG.eliminate_start(grammar)
```

--------------------------------

### ChartParserApp Setup and Execution

Source: https://www.nltk.org/_modules/nltk/app/chartparser_app.html

Defines a context-free grammar, a sample sentence, and then initializes and runs the ChartParserApp. This is the main entry point for demonstrating the chart parsing functionality.

```python
def app():
    grammar = CFG.fromstring(
        """
    # Grammatical productions.
        S -> NP VP
        VP -> VP PP | V NP | V
        NP -> Det N | NP PP
        PP -> P NP
    # Lexical productions.
        NP -> 'John' | 'I'
        Det -> 'the' | 'my' | 'a'
        N -> 'dog' | 'cookie' | 'table' | 'cake' | 'fork'
        V -> 'ate' | 'saw'
        P -> 'on' | 'under' | 'with'
    """
    )

    sent = "John ate the cake on the table with a fork"
    sent = "John ate the cake on the table"
    tokens = list(sent.split())

    print("grammar= (")
    for rule in grammar.productions():
        print(( "    ", repr(rule) + ","))
    print(")")
    print("tokens = %r" % tokens)
    print('Calling "ChartParserApp(grammar, tokens)"...')
    ChartParserApp(grammar, tokens).mainloop()
```

--------------------------------

### Get token spans with NLTKWordTokenizer

Source: https://www.nltk.org/api/nltk.tokenize.destructive.html

Use span_tokenize to get the start and end indices of tokens in a string. This method assumes the text is already sentence-segmented.

```python
from nltk.tokenize import NLTKWordTokenizer
s = '''Good muffins cost $3.88\nin New (York).  Please (buy) me\ntwo of them.\n(Thanks).'''
expected = [(0, 4), (5, 12), (13, 17), (18, 19), (19, 23),
(24, 26), (27, 30), (31, 32), (32, 36), (36, 37), (37, 38),
(40, 46), (47, 48), (48, 51), (51, 52), (53, 55), (56, 59),
(60, 62), (63, 68), (69, 70), (70, 76), (76, 77), (77, 78)]
list(NLTKWordTokenizer().span_tokenize(s)) == expected
```

```python
expected = ['Good', 'muffins', 'cost', '$', '3.88', 'in',
'New', '(', 'York', ')', '.', 'Please', '(', 'buy', ')',
'me', 'two', 'of', 'them.', '(', 'Thanks', ')', '.']
[s[start:end] for start, end in NLTKWordTokenizer().span_tokenize(s)] == expected
```

--------------------------------

### Gensim Doctest Example

Source: https://www.nltk.org/_sources/howto/gensim.rst.txt

This snippet is a doctest example for Gensim, typically used for testing and demonstrating basic functionalities. Ensure Gensim is installed to run these tests.

```python
>>> from gensim.corpora import Dictionary
>>> from gensim.models import LdaModel
>>> from gensim.parsing.preprocessing import preprocess_string, strip_tags, strip_punctuation, strip_numeric, remove_stopwords, split_alphanum, strip_short
>>> texts = ["This is the first document.", "This document is the second document.", "And this is the third one.", "Is this the first document?"]
>>> processed_texts = [preprocess_string(text) for text in texts]
>>> dictionary = Dictionary(processed_texts)
>>> corpus = [dictionary.doc2bow(text) for text in processed_texts]
>>> lda = LdaModel(corpus=corpus, id2word=dictionary, num_topics=2)
>>> lda.print_topics()
[(0, '0.068*''), 1.0), (1, '0.068*''), 1.0)]
```

--------------------------------

### Initialize Senna Pipeline and Tag Sentence

Source: https://www.nltk.org/api/nltk.classify.senna.html

Demonstrates how to initialize the Senna pipeline with specified operations and tag a sample sentence. Ensure the SENNA executable path is correct.

```python
from nltk.classify import Senna
pipeline = Senna('/usr/share/senna-v3.0', ['pos', 'chk', 'ner'])
sent = 'Dusseldorf is an international business center'.split()
[(token['word'], token['chk'], token['ner'], token['pos']) for token in pipeline.tag(sent)]
```

--------------------------------

### Boxer Installation and Setup

Source: https://www.nltk.org/api/nltk.sem.boxer.html

Before using Boxer, ensure the environment variable CANDC is set to the bin directory of your CandC installation. The models directory should be in the CandC root directory.

```shell
/path/to/candc/
bin/
    candc
    boxer
models/
    boxer/

```

--------------------------------

### Initialize Key Bindings

Source: https://www.nltk.org/_modules/nltk/sem/drt_glue_demo.html

Sets up keyboard shortcuts for common actions like quitting, navigating to the next/previous item, and exiting.

```python
    def _init_bindings(self):
        # Key bindings are a good thing.
        self._top.bind("<Control-q>", self.destroy)
        self._top.bind("<Control-x>", self.destroy)
        self._top.bind("<Escape>", self.destroy)
        self._top.bind("n", self.next)
        self._top.bind("<space>", self.next)
        self._top.bind("p", self.prev)
        self._top.bind("<BackSpace>", self.prev)
```

--------------------------------

### Configure and Run Prover9 Command

Source: https://www.nltk.org/_modules/nltk/inference/prover9.html

Demonstrates how to instantiate and run the Prover9Command, including setting executable path and search parameters. Useful for testing prover configuration.

```python
def test_config():
    a = Expression.fromstring("(walk(j) & sing(j))")
    g = Expression.fromstring("walk(j)")
    p = Prover9Command(g, assumptions=[a])
    p._executable_path = None
    p.prover9_search = []
    p.prove()
    # config_prover9('/usr/local/bin')
    print(p.prove())
    print(p.proof())
```

--------------------------------

### Instantiate and Use ResolutionProverCommand

Source: https://www.nltk.org/howto/inference.html

Demonstrates initializing a ResolutionProverCommand with assumptions, attempting to get a proof before calling prove(), executing the proof, and retrieving the proof string. Also shows calling prove() multiple times and accessing assumptions and goal.

```python
>>> prover = ResolutionProverCommand(c, [p1,p2])
>>> print(prover.proof())
Traceback (most recent call last):
  File "...", line 1212, in __run
    compileflags, 1) in test.globs
  File "<doctest nltk/test/inference.doctest[10]>", line 1, in <module>
  File "...", line ..., in proof
    raise LookupError("You have to call prove() first to get a proof!")
LookupError: You have to call prove() first to get a proof!
>>> prover.prove()
True
>>> print(prover.proof())
[1] {-mortal(socrates)}     A
[2] {man(socrates)}         A
[3] {-man(z4), mortal(z4)}  A
[4] {-man(socrates)}        (1, 3)
[5] {mortal(socrates)}      (2, 3)
[6] {}                      (1, 5)


```

```python
>>> prover.prove()
True
>>> prover.prove()
True

```

```python
>>> prover.assumptions()
[<ApplicationExpression man(socrates)>, <AllExpression all x.(man(x) -> mortal(x))>]
```

```python
>>> prover.goal()
<ApplicationExpression mortal(socrates)>
```

```python
>>> prover.retract_assumptions([read_expr('man(socrates)')])
>>> print(prover.proof())
Traceback (most recent call last):
  File "...", line 1212, in __run
    compileflags, 1) in test.globs
  File "<doctest nltk/test/inference.doctest[10]>", line 1, in <module>
  File "...", line ..., in proof
    raise LookupError("You have to call prove() first to get a proof!")
LookupError: You have to call prove() first to get a proof!
>>> prover.prove()
False
>>> print(prover.proof())
[1] {-mortal(socrates)}     A
[2] {-man(z6), mortal(z6)}  A
[3] {-man(socrates)}        (1, 2)

>>> prover.add_assumptions([read_expr('man(socrates)')])
>>> prover.prove()
True

```

--------------------------------

### Get Synset Definition and Examples with NLTK WordNet

Source: https://www.nltk.org/book/ch02.html

Access the human-readable definition of a synset using `synset.definition()` and example sentences using `synset.examples()`. These provide context and illustrate the usage of the synset.

```python
wn.synset('car.n.01').definition()
```

```python
wn.synset('car.n.01').examples()
```

--------------------------------

### Access TIMIT Word Timings

Source: https://www.nltk.org/book/ch11.html

Get the start and end times for each word in a TIMIT corpus utterance. The `word_times()` method returns a list of tuples, each containing a word, its start time, and its end time.

```python
>>> nltk.corpus.timit.word_times('dr1-fvmh0/sa1')
[('she', 7812, 10610), ('had', 10610, 14496), ('your', 14496, 15791),
('dark', 15791, 20720), ('suit', 20720, 25647), ('in', 25647, 26906),
('greasy', 26906, 32668), ('wash', 32668, 37890), ('water', 38531, 42417),
('all', 43091, 46052), ('year', 46052, 50522)]
```

--------------------------------

### Vocabulary Initialization and Lookup Examples

Source: https://www.nltk.org/api/nltk.lm.vocabulary.html

Provides examples of initializing a Vocabulary with specific words and an unknown cutoff, then demonstrates the lookup method for single words and sequences, including nested sequences.

```python
>>> from nltk.lm import Vocabulary
>>> vocab = Vocabulary(["a", "b", "c", "a", "b"], unk_cutoff=2)
>>> vocab.lookup("a")
'a'
>>> vocab.lookup("aliens")
'<UNK>'
>>> vocab.lookup(["a", "b", "c", ["x", "b"]])
('a', 'b', '<UNK>', ('<UNK>', 'b'))
```

--------------------------------

### StanfordSegmenter Initialization and Usage Example

Source: https://www.nltk.org/_modules/nltk/tokenize/stanford_segmenter.html

Demonstrates how to initialize the StanfordSegmenter for Chinese and Arabic, and then segment sample sentences. Note that these examples are marked with SKIP as they require external Stanford NLP models and Java setup.

```python
from nltk.tokenize.stanford_segmenter import StanfordSegmenter
seg = StanfordSegmenter() # doctest: +SKIP
seg.default_config('zh') # doctest: +SKIP
sent = u'这是斯坦福中文分词器测试'
print(seg.segment(sent)) # doctest: +SKIP

seg.default_config('ar') # doctest: +SKIP
sent = u'هذا هو تصنيف ستانفورد العربي للكلمات'
print(seg.segment(sent.split())) # doctest: +SKIP
```

--------------------------------

### Create and Run CFG Demo

Source: https://www.nltk.org/_modules/nltk/draw/cfg.html

Defines a sample context-free grammar and text, then creates and runs a CFG demo. Imports necessary classes from NLTK.

```python
def demo2():
    from nltk import CFG, Nonterminal, Production

    nonterminals = "S VP NP PP P N Name V Det"
    (S, VP, NP, PP, P, N, Name, V, Det) = (Nonterminal(s) for s in nonterminals.split())
    productions = (
        # Syntactic Productions
        Production(S, [NP, VP]),
        Production(NP, [Det, N]),
        Production(NP, [NP, PP]),
        Production(VP, [VP, PP]),
        Production(VP, [V, NP, PP]),
        Production(VP, [V, NP]),
        Production(PP, [P, NP]),
        Production(PP, []),
        Production(PP, ["up", "over", NP]),
        # Lexical Productions
        Production(NP, ["I"]),
        Production(Det, ["the"]),
        Production(Det, ["a"]),
        Production(N, ["man"]),
        Production(V, ["saw"]),
        Production(P, ["in"]),
        Production(P, ["with"]),
        Production(N, ["park"]),
        Production(N, ["dog"]),
        Production(N, ["statue"]),
        Production(Det, ["my"]),
    )
    grammar = CFG(S, productions)

    text = "I saw a man in the park".split()
    d = CFGDemo(grammar, text)
    d.mainloop()
```

--------------------------------

### Get Timit Audio Data

Source: https://www.nltk.org/api/nltk.corpus.reader.html

Retrieve audio data for a specified utterance. Optionally specify start and end times.

```python
audiodata(_utterance_ , _start =0_, _end =None_)
```

--------------------------------

### CoreNLPServer Initialization and Usage

Source: https://www.nltk.org/_modules/nltk/parse/corenlp.html

Demonstrates how to initialize and manage a CoreNLP server. The server can be started and stopped, and its lifecycle can be managed using a context manager.

```APIDOC
## CoreNLPServer

Manages the lifecycle of a Stanford CoreNLP server.

### Description

This class provides functionality to start, stop, and manage a Stanford CoreNLP server. It handles finding the necessary JAR files, configuring Java options, and ensuring the server is running and ready.

### Methods

#### `__init__`

Initializes the `CoreNLPServer` with specified paths, verbosity, Java options, and CoreNLP specific options.

- **`path_to_jar`** (str, optional): Path to the CoreNLP JAR file.
- **`path_to_models_jar`** (str, optional): Path to the CoreNLP models JAR file.
- **`verbose`** (bool, optional): If True, enables verbose output.
- **`java_options`** (list[str], optional): List of Java options to configure the JVM.
- **`corenlp_options`** (list[str], optional): List of options to configure the CoreNLP server.
- **`port`** (int, optional): The port to run the server on. If not specified, a free port is chosen.

#### `start`

Starts the CoreNLP server.

- **`stdout`** (str): Specifies where CoreNLP output is redirected. Valid values are 'devnull', 'stdout', 'pipe'.
- **`stderr`** (str): Specifies where CoreNLP error output is redirected. Valid values are 'devnull', 'stdout', 'pipe'.

#### `stop`

Stops the CoreNLP server.

#### `__enter__`

Enters the runtime context related to this object. Starts the server.

#### `__exit__`

Exits the runtime context related to this object. Stops the server.

### Example Usage

```python
from nltk.parse import CoreNLPParser

# Assuming CoreNLP server is running on default port
parser = CoreNLPParser()

# Or start a server instance and use it
# with CoreNLPServer() as server:
#     parser = CoreNLPParser(url=server.url)
#     # ... use parser ...
```

```
Tokens:
Raw