### Pyate Installation Details

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/docs/_build/html/_sources/installation.rst.txt

Example output from 'pip show pyate' indicating a successful installation.

```bash
Name: pyate
Version: 0.4.2
Summary: PYthon Automated Term Extraction
Home-page: https://github.com/kevinlu1248/pyate
Author: Kevin Lu
Author-email: kevinlu1248@gmail.com
License: MIT
Location: /home/kevin/.local/lib/python3.8/site-packages
Requires: numpy, pyahocorasick, pandas, spacy
Required-by:
```

--------------------------------

### Install Pyate with pip

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/installation.rst.txt

Install Pyate and a spaCy model using pip. Ensure you have pip installed.

```bash
pip install pyate https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz
```

--------------------------------

### Install PyATE and spaCy Model

Source: https://context7.com/kevinlu1248/pyate/llms.txt

Install the PyATE library and download the small English spaCy language model for POS tagging.

```bash
pip install pyate
python -m spacy download en_core_web_sm
```

--------------------------------

### Install PyATE and spaCy model

Source: https://github.com/kevinlu1248/pyate/blob/master/README.md

Install the package via pip and download the required spaCy language model.

```bash
pip install pyate 
spacy download en_core_web_sm
```

--------------------------------

### Verify Pyate Installation

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/installation.rst.txt

Check if Pyate has been installed correctly by using pip show. This command displays information about the installed package.

```bash
pip show pyate
```

--------------------------------

### Build Pyate from Source

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/installation.rst.txt

Clone the Pyate repository and install it locally. This method is useful for development or if you need the latest unreleased version.

```bash
git clone https://github.com/kevinlu1248/pyate.git
# or alternatively, using the gh cli
# gh repo clone kevinlu1248/pyate
pip install pyate
```

--------------------------------

### Example Usage of combo_basic

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_modules/pyate/combo_basic.html

Demonstrates how to use the combo_basic algorithm with a sample corpus. It prints the top 50 extracted terms sorted by their scores.

```python
if __name__ == "__main__":
    corpus = "Hello I am a term extractor."
    print(TermExtraction(corpus).combo_basic().sort_values(ascending=False).head(50))
```

--------------------------------

### Example Usage of TermExtraction

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_modules/pyate/term_extraction.html

Demonstrates how to initialize TermExtraction with a corpus and call count_terms_from_documents with verbose output.

```python
if __name__ == "__main__":
    PATH_TO_GENERAL_DOMAIN = "../data/wiki_testing.pkl"
    PATH_TO_TECHNICAL_DOMAIN = "../data/pmc_testing.pkl"
    wiki = pd.read_pickle(PATH_TO_GENERAL_DOMAIN)
    pmc = pd.read_pickle(PATH_TO_TECHNICAL_DOMAIN)
    print(
        TermExtraction(pmc[:100]).count_terms_from_documents(
            seperate=True, verbose=True
        )
    )
```

--------------------------------

### Term Extractor Example Usage

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_modules/pyate/term_extractor.html

Demonstrates how to use the term_extractor function with specific technical and general corpora, including verbose output. This example loads data from pickle files.

```python
if __name__ == "__main__":
    PATH_TO_GENERAL_DOMAIN = "../data/wiki_testing.pkl"
    PATH_TO_TECHNICAL_DOMAIN = "../data/pmc_testing.pkl"
    wiki = pd.read_pickle(PATH_TO_GENERAL_DOMAIN)
    pmc = pd.read_pickle(PATH_TO_TECHNICAL_DOMAIN)
    # term_extractor(pmc[:50], wiki[:250])

    # print(term_extractor(pmc, wiki).sort_values(ascending=False).head(50))
    print(
        term_extractor(pmc[:50], wiki[:1000], verbose=True)
    )
```

--------------------------------

### Install PyATE spaCy Model

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/models.rst.txt

Install the en_acl_terms_sm spaCy model for term extraction. This model is not included with PyATE by default.

```bash
pip install https://github.com/kevinlu1248/pyate/releases/download/v0.4.2/en_acl_terms_sm-2.0.4.tar.gz
```

--------------------------------

### Example usage of cvalues

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_modules/pyate/cvalues.html

This is a simple example demonstrating how to use the cvalues function with a basic corpus string. It prints the extracted terms and their calculated c-values.

```python
if __name__ == "__main__":
    corpus = "Hello, I am a term extractor."
    print(TermExtraction(corpus).cvalues(verbose=True))
```

--------------------------------

### Example Usage of Basic Term Extraction

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/docs/_build/html/_modules/pyate/basic.html

Demonstrates how to use the basic term extraction method with a sample corpus and prints the top 50 extracted terms.

```python
if __name__ == "__main__":
    corpus = "Hello I am a term extractor."
    print(TermExtraction(corpus).basic().sort_values(ascending=False).head(50))
```

--------------------------------

### Load General English Corpus with Pandas

Source: https://github.com/kevinlu1248/pyate/blob/master/README.md

Reads a general English corpus from a zip file located in the site-packages directory using Pandas. Ensure Pyate is installed.

```python
import pandas as pd
from distutils.sysconfig import get_python_lib
df = pd.read_csv(get_python_lib() + "/pyate/default_general_domain.en.zip")["SECTION_TEXT"]
print(df.head())
```

--------------------------------

### Example Usage of Weirdness Function

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_modules/pyate/weirdness.html

Demonstrates how to use the 'weirdness' function with sample data loaded from pickle files. Prints the top 50 terms sorted by their calculated weirdness score.

```python
if __name__ == "__main__":
    PATH_TO_GENERAL_DOMAIN = "../data/wiki_testing.pkl"
    PATH_TO_TECHNICAL_DOMAIN = "../data/pmc_testing.pkl"
    wiki = pd.read_pickle(PATH_TO_GENERAL_DOMAIN)
    pmc = pd.read_pickle(PATH_TO_TECHNICAL_DOMAIN)
    pairdf = weirdness(pmc[:200], wiki[:500])
    print(pairdf.sort_values(ascending=False).head(50))
```

--------------------------------

### PyATE Algorithms Overview

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/algorithms.rst.txt

This section provides an overview of the available algorithms and their parameters. The table indicates which parameters are supported by each algorithm.

```APIDOC
## PyATE Algorithms

Five algorithms are implemented in this package:

* C-Value
* Basic
* Combo Basic
* Weirdness
* Term Extractor.

### General Parameters

* ``technical_corpus: Union[str, Sequence[str]]``: The corpus to search for technical terms. If a string, the algorithm runs on that corpus. If a sequence of strings, it runs on each element.
* ``general_corpus: pd.Series = TermExtraction.get_general_domain()``: The corpus to compare against. Defaults to a general domain corpus.
* ``general_corpus_size: int = 300``: The number of sentences desired for the ``general_corpus``.
* ``verbose: bool = False``: Whether to display a progress bar.
* ``weights``: Algorithm-specific weights for determining term likelihood.
* ``smoothing: double = 0.01``: A factor added before division to prevent division by zero errors.
* ``have_single_word: bool = False``: Determines if single words are considered as terms.
* ``technical_counts: Mapping[str, int]``: A frequency counter for candidates. If not provided, it will be computed.
* ``threshold: float = 0``: The threshold for C-Value algorithms.

### Algorithm Parameter Support

+-----------------+--------------------------+------------------------+-----------------------------+--------------------+-----------------+-----------------+-------------------+--------------------------+--------------------------+-------------------+
|                 | ``technical_corpus`` | ``general_corpus`` | ``general_corpus_size`` | ``normalized`` | ``verbose`` | ``weights`` | ``smoothing`` | ``have_single_word`` | ``technical_counts`` | ``threshold`` |
+-----------------+--------------------------+------------------------+-----------------------------+--------------------+-----------------+-----------------+-------------------+--------------------------+--------------------------+-------------------+
| C-Value         | ✓                        |                        |                             |                    | ✓               |                 | ✓                 | ✓                        |                          | ✓                 |
+-----------------+--------------------------+------------------------+-----------------------------+--------------------+-----------------+-----------------+-------------------+--------------------------+--------------------------+-------------------+
| Basic           | ✓                        |                        |                             |                    | ✓               | ✓               | ✓                 | ✓                        | ✓                        |                   |
+-----------------+--------------------------+------------------------+-----------------------------+--------------------+-----------------+-----------------+-------------------+--------------------------+--------------------------+-------------------+
| Combo Basic     | ✓                        |                        |                             |                    | ✓               | ✓               | ✓                 | ✓                        | ✓                        |                   |
+-----------------+--------------------------+------------------------+-----------------------------+--------------------+-----------------+-----------------+-------------------+--------------------------+--------------------------+-------------------+
| Weirdness       | ✓                        | ✓                      | ✓                           | ✓                  | ✓               |                 |                   |                          | ✓                        |                   |
+-----------------+--------------------------+------------------------+-----------------------------+--------------------+-----------------+-----------------+-------------------+--------------------------+--------------------------+-------------------+
| Term Extractor  | ✓                        | ✓                      | ✓                           |                    | ✓               | ✓               |                   |                          | ✓                        |                   |
+-----------------+--------------------------+------------------------+-----------------------------+--------------------+-----------------+-----------------+-------------------+--------------------------+--------------------------+-------------------+

**Note**: ``technical_corpus`` is **required** for all algorithms.
```

--------------------------------

### Import PyATE algorithm

Source: https://github.com/kevinlu1248/pyate/blob/master/README.md

Import the combo_basic algorithm to begin term extraction.

```python
from pyate import combo_basic
```

--------------------------------

### pyate.combo_basic Module

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/pyate.rst.txt

Documentation for the pyate.combo_basic module, related to combination logic.

```APIDOC
## pyate.combo_basic Module

### Description
This module handles basic combination logic within the pyate package.

### Members
All members of this module are documented, including undocumented ones. Inheritance is also shown.
```

--------------------------------

### GET /count_terms_from_document

Source: https://github.com/kevinlu1248/pyate/blob/master/html/pyate.html

Counts the frequency of each term in a single document.

```APIDOC
## GET /count_terms_from_document

### Description
Counts the frequency of each term in the class and returns a default dict mapping each string to the number of occurrences.

### Method
GET

### Parameters
#### Query Parameters
- **document** (str) - Required - The document string to analyze.
```

--------------------------------

### GET /count_terms_from_documents

Source: https://github.com/kevinlu1248/pyate/blob/master/html/pyate.html

Counts terms from documents and returns a pandas Series.

```APIDOC
## GET /count_terms_from_documents

### Description
Counts terms from the documents. If the corpus is an iterable of strings, it processes each string. Returns a pandas Series or an iterable of default dicts depending on the 'seperate' parameter.

### Method
GET

### Parameters
#### Query Parameters
- **seperate** (bool) - Optional - If true, returns an iterable of default dicts; otherwise, returns a single default dict with the sum of frequencies.
- **verbose** (bool) - Optional - Enables verbose output.
```

--------------------------------

### Basic Algorithm

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/algorithms.rst.txt

Documentation for the Basic algorithm, including its parameters and usage.

```APIDOC
## POST /pyate/basic

### Description

Applies the Basic algorithm for term extraction.

### Method

POST

### Endpoint

/pyate/basic

### Parameters

#### Query Parameters

* **technical_corpus** (Union[str, Sequence[str]]) - Required - The corpus to search for technical terms.
* **general_corpus** (pd.Series) - Optional - The corpus to compare against. Defaults to TermExtraction.get_general_domain().
* **general_corpus_size** (int) - Optional - The number of sentences desired for the general corpus. Defaults to 300.
* **verbose** (bool) - Optional - Whether to display a progress bar. Defaults to False.
* **weights** - Optional - Algorithm-specific weights.
* **smoothing** (double) - Optional - Smoothing factor to prevent division by zero. Defaults to 0.01.
* **have_single_word** (bool) - Optional - Whether to consider single words as terms. Defaults to False.
* **technical_counts** (Mapping[str, int]) - Optional - Pre-computed frequency counts for technical terms.
* **threshold** (float) - Optional - Threshold for term extraction. Defaults to 0.

### Request Example

```json
{
  "technical_corpus": "This is a technical document about machine learning.",
  "general_corpus": null, 
  "general_corpus_size": 300,
  "verbose": true,
  "weights": {"term1": 0.5, "term2": 0.5},
  "smoothing": 0.01,
  "have_single_word": true,
  "technical_counts": null,
  "threshold": 0.5
}
```

### Response

#### Success Response (200)

- **terms** (list[str]) - A list of extracted terms.

#### Response Example

```json
{
  "terms": ["machine learning", "algorithm"]
}
```
```

--------------------------------

### Pyate Package Overview

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/pyate.rst.txt

This section provides an overview of the pyate package and its submodules.

```APIDOC
## Pyate Package

### Description
The pyate package is a Python library designed for various text processing tasks, including term extraction and analysis.

### Module Contents
This package contains the following submodules:
- `pyate.basic`
- `pyate.combo_basic`
- `pyate.cvalues`
- `pyate.term_extraction`
- `pyate.term_extraction_pipeline`
- `pyate.term_extractor`
- `pyate.weirdness`

### Members
All members of the `pyate` module are accessible, including those from its submodules.
```

--------------------------------

### pyate.basic Module

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/pyate.rst.txt

Documentation for the pyate.basic module, which contains fundamental functionalities.

```APIDOC
## pyate.basic Module

### Description
This module provides basic functionalities for the pyate package.

### Members
All members of this module are documented, including undocumented ones. Inheritance is also shown.
```

--------------------------------

### Get General Domain Data

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/docs/_build/html/_modules/pyate/term_extraction.html

Fetches a pandas Series of general domain data for a specified language and size. Caches the data for subsequent calls.

```python
@staticmethod
def get_general_domain(language: str = None, size: int = None):
    """
    For getting a pandas Series of domains.
    """
    if language is None:
        language = TermExtraction.config["language"]
    if size is None:
        size = TermExtraction.config["DEFAULT_GENERAL_DOMAIN_SIZE"]
    if (language, size) not in TermExtraction.DEFAULT_GENERAL_DOMAINS:
        TermExtraction.DEFAULT_GENERAL_DOMAINS[(language, size)] = pd.read_csv(
            pkg_resources.resource_stream(
                __name__,
                f"default_general_domain.{language}.zip",
            ),
            nrows=size,
        )
    return TermExtraction.DEFAULT_GENERAL_DOMAINS[(language, size)]

```

--------------------------------

### Extract Terms with PyATE Model

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_sources/models.rst.txt

Load the installed spaCy model and use it to extract terms from a given text. The extracted terms are stored in the doc.ents attribute.

```python
import spacy

nlp = spacy.load("en_acl_terms_sm")
doc = nlp("Hello world, I am a term extraction algorithm.")
print(doc.ents)

```

--------------------------------

### pyate.cvalues Module

Source: https://github.com/kevinlu1248/pyate/blob/master/html/_sources/pyate.rst.txt

Documentation for the pyate.cvalues module.

```APIDOC
## pyate.cvalues Module

### Description
This module deals with cvalues.

### Members
- **__all__** (list)
- **__builtins__** (dict)
- **__cached__** (str)
- **__doc__** (str)
- **__file__** (str)
- **__loader__** (module_loader)
- **__name__** (str)
- **__package__** (str)
- **__spec__** (module_spec)
- **get_cvalues** (function)
- **get_cvalues_from_text** (function)
```

--------------------------------

### Extract Terms with spaCy Model

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/models.html

Load the installed spaCy model and use it to extract terms from a given text. The extracted terms are available in the 'doc.ents' attribute.

```python
import spacy
nlp = spacy.load("en_acl_terms_sm")
doc = nlp("Hello world, I am a term extraction algorithm.")
print(doc.ents)
```

--------------------------------

### pyate.combo_basic Module

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/pyate.html

Implements combined basic methods for term extraction.

```APIDOC
## pyate.combo_basic.combo_basic

### Description
Performs term extraction using combined basic methods.

### Method
`combo_basic`

### Parameters
- **technical_corpus** (Union[str, Sequence[str]]) - The technical corpus or a sequence of documents.
- **smoothing** (float) - Smoothing factor. Defaults to 0.01.
- **verbose** (bool) - If True, enables verbose output. Defaults to False.
- **have_single_word** (bool) - If True, considers single words as terms. Defaults to False.
- **technical_counts** (Mapping[str, int]) - Pre-computed technical term counts. Defaults to None.
- **weights** (Sequence[float]) - Weights for combining different methods. Defaults to None.
```

```APIDOC
## pyate.combo_basic.helper_get_subsequences

### Description
Helper function to get subsequences of a string.

### Method
`helper_get_subsequences`

### Parameters
- **s** (str) - The input string.

### Returns
- List[str] - A list of subsequences.
```

--------------------------------

### Extract Terms using spaCy Model

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/models.rst.txt

Load the installed spaCy model and use it to extract terms from a given document. The extracted terms are stored in doc.ents.

```python
import spacy

nlp = spacy.load("en_acl_terms_sm")
doc = nlp("Hello world, I am a term extraction algorithm.")
print(doc.ents)
"""
(term extraction, algorithm)
"""
```

--------------------------------

### Algorithm Parameters

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/algorithms.md

General parameters applicable to most algorithms in the PyATE package.

```APIDOC
## General Parameters

These parameters are common across several algorithms within the PyATE package.

* `technical_corpus` (Union[str, Sequence[str]]): Required. The corpus from which technical terms are to be searched. If a string, it's treated as a single corpus. If a sequence of strings, the algorithm runs on each element.
* `general_corpus` (pd.Series, optional): The corpus used for comparison. Defaults to `TermExtraction.get_general_domain()`.
* `general_corpus_size` (int, optional): The desired number of sentences for the `general_corpus`. Defaults to 300.
* `verbose` (bool, optional): If True, displays a progress bar. Defaults to False.
* `weights` (list[float], optional): Algorithm-specific weights influencing the importance of different factors in term scoring.
* `smoothing` (double, optional): A factor added before division to prevent division by zero errors. Defaults to 0.01.
* `have_single_word` (bool, optional): Determines if single words should be considered as potential terms. Defaults to False.
* `technical_counts` (Mapping[str, int], optional): A frequency counter mapping candidate terms to their frequencies. If not provided, it will be computed. Useful for optimizing multiple algorithm runs.
* `threshold` (float, optional): A threshold specific to C-Value algorithms. Defaults to 0.
```

--------------------------------

### Initialize TermExtractionPipeline

Source: https://github.com/kevinlu1248/pyate/blob/master/html/_modules/pyate/term_extraction_pipeline.html

Initializes the TermExtractionPipeline, setting up the term extraction function, arguments, and spaCy's Matcher. It also sets a spaCy Doc extension for storing extracted terms.

```python
class TermExtractionPipeline:
    """
    This is for adding PyATE as a spaCy pipeline component.
    """
    def __init__(
        self,
        nlp,
        func: Callable[..., pd.Series] = combo_basic,
        force: bool = True,
        *args,
        **kwargs
    ) -> None:
        """
        This is for initializing the TermExtractionPipeline.
        """
        self.func = func
        self.args = args
        self.kwargs = kwargs
        self.__name__ = self.func.__name__
        self.matcher = Matcher(nlp.vocab)
        Doc.set_extension(self.__name__, default=None, force=force)
        self.term_counter = None

        def add_to_counter(matcher, doc, i, matches) -> Doc:
            match_id, start, end = matches[i]
            candidate = str(doc[start:end])
            if (
                TermExtraction.word_length(candidate)
                <= TermExtraction.config["MAX_WORD_LENGTH"]
            ):
                self.term_counter[candidate] += 1

        for i, pattern in enumerate(TermExtraction.patterns):
            self.matcher.add("term{}".format(i), add_to_counter, pattern)
```

--------------------------------

### Get spaCy NLP Model

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/docs/_build/html/_modules/pyate/term_extraction.html

Retrieves a spaCy NLP model for a given language. Caches models for reuse. Defaults to the configured language if none is specified.

```python
@staticmethod
def get_nlp(language: str = None):
    """
    For getting the spaCy nlp.
    """
    if language is None:
        language = TermExtraction.language
    if language not in TermExtraction.nlps:
        TermExtraction.nlps[language] = spacy.load(
            TermExtraction.config["spacy_model"], parser=False, entity=False
        )
    return TermExtraction.nlps[language]

```

--------------------------------

### Get spaCy NLP Model

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_modules/pyate/term_extraction.html

Static method to retrieve or load a spaCy NLP model for a given language. It caches loaded models to improve performance.

```python
    @staticmethod
    def get_nlp(language: str = None):
        """
        For getting the spaCy nlp.
        """
        if language is None:
            language = TermExtraction.language
        if language not in TermExtraction.nlps:
            TermExtraction.nlps[language] = spacy.load(
                TermExtraction.config["spacy_model"], parser=False, entity=False
            )
        return TermExtraction.nlps[language]

```

--------------------------------

### pyate.basic.basic

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/docs/_build/html/pyate.html

Initializes and performs basic term extraction on a technical corpus.

```APIDOC
## pyate.basic.basic

### Description
Performs basic term extraction on a given technical corpus.

### Method
Function

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
None
```

--------------------------------

### Initialize TermExtractionPipeline

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_modules/pyate/term_extraction_pipeline.html

Initializes the TermExtractionPipeline, setting up the function, arguments, and spaCy pipeline component extension. It configures a spaCy Matcher with predefined patterns for term extraction.

```python
def __init__(
        self,
        nlp,
        func: Callable[..., pd.Series] = combo_basic,
        force: bool = True,
        *args,
        **kwargs
    ) -> None:
        """
        This is for initializing the TermExtractionPipeline.
        """
        self.func = func
        self.args = args
        self.kwargs = kwargs
        self.__name__ = self.func.__name__
        self.matcher = Matcher(nlp.vocab)
        Doc.set_extension(self.__name__, default=None, force=force)
        self.term_counter = None

        def add_to_counter(matcher, doc, i, matches) -> Doc:
            match_id, start, end = matches[i]
            candidate = str(doc[start:end])
            if (
                TermExtraction.word_length(candidate)
                <= TermExtraction.config["MAX_WORD_LENGTH"]
            ):
                self.term_counter[candidate] += 1

        for i, pattern in enumerate(TermExtraction.patterns):
            self.matcher.add("term{}".format(i), add_to_counter, pattern)
```

--------------------------------

### Sort and Get Top 50 Values

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/docs/_build/html/_modules/pyate/term_extractor.html

Sorts a DataFrame by values in descending order and retrieves the top 50 entries. This is useful for identifying the most frequent or significant items.

```python
.sort_values(ascending=False)
        .head(50)
```

--------------------------------

### Helper function to get subsequences

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_modules/pyate/cvalues.html

This function splits a string into words and generates all possible contiguous subsequences, excluding the entire sequence itself. Useful for identifying nested terms.

```python
import math
from typing import List
from typing import Mapping

import pandas as pd
from tqdm import tqdm

from .term_extraction import add_term_extraction_method
from .term_extraction import Corpus
from .term_extraction import TermExtraction

def helper_get_subsequences(s: str) -> List[str]:
    sequence = s.split()
    if len(sequence) <= 2:
        return []
    answer = []
    for left in range(len(sequence) + 1):
        for right in range(left + 1, len(sequence) + 1):
            if left == 0 and right == len(sequence):
                continue
            answer.append(" ".join(sequence[left:right]))
    return answer
```

--------------------------------

### pyate.cvalues Module

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/pyate.rst.txt

Documentation for the pyate.cvalues module, dealing with custom values.

```APIDOC
## pyate.cvalues Module

### Description
This module is responsible for managing custom values in the pyate package.

### Members
All members of this module are documented, including undocumented ones. Inheritance is also shown.
```

--------------------------------

### Get General Domain Data

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_modules/pyate/term_extraction.html

Static method to retrieve a pandas Series of general domain terms. It loads data from CSV files and caches it based on language and size.

```python
    @staticmethod
    def get_general_domain(language: str = None, size: int = None):
        """
        For getting a pandas Series of domains.
        """
        if language is None:
            language = TermExtraction.config["language"]
        if size is None:
            size = TermExtraction.config["DEFAULT_GENERAL_DOMAIN_SIZE"]
        if (language, size) not in TermExtraction.DEFAULT_GENERAL_DOMAINS:
            TermExtraction.DEFAULT_GENERAL_DOMAINS[(language, size)] = pd.read_csv(
                pkg_resources.resource_stream(
                    __name__,
                    f"default_general_domain.{language}.csv",
                ),
                nrows=size,
            )
        return TermExtraction.DEFAULT_GENERAL_DOMAINS[(language, size)]

```

--------------------------------

### Helper function to get subsequences

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_modules/pyate/cvalues.html

This function splits a string into words and generates all possible contiguous subsequences, excluding the entire string itself. It's used internally by the cvalues algorithm.

```python
def helper_get_subsequences(s: str) -> List[str]:
    sequence = s.split()
    if len(sequence) <= 2:
        return []
    answer = []
    for left in range(len(sequence) + 1):
        for right in range(left + 1, len(sequence) + 1):
            if left == 0 and right == len(sequence):
                continue
            answer.append(" ".join(sequence[left:right]))
    return answer
```

--------------------------------

### POST /configure

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_modules/pyate/term_extraction.html

Updates the global configuration settings for the TermExtraction class, including language, spaCy model, and word length constraints.

```APIDOC
## POST /configure

### Description
Updates the configuration settings used by the TermExtraction class.

### Method
POST

### Endpoint
/configure

### Request Body
- **new_settings** (Dict[str, Any]) - Required - A dictionary containing configuration keys such as 'spacy_model', 'language', 'MAX_WORD_LENGTH', 'DEFAULT_GENERAL_DOMAIN_SIZE', and 'dtype'.
```

--------------------------------

### PyATE Module Index

Source: https://github.com/kevinlu1248/pyate/blob/master/html/genindex.html

This section lists the available modules within the PyATE library and provides links to their respective documentation.

```APIDOC
## PyATE Modules

This index provides an overview of the modules available in the PyATE library.

### Modules

*   [pyate](pyate.html)
*   [pyate.basic](pyate.html#module-pyate.basic)
*   [pyate.combo_basic](pyate.html#module-pyate.combo_basic)
*   [pyate.cvalues](pyate.html#module-pyate.cvalues)
*   [pyate.term_extraction](pyate.html#module-pyate.term_extraction)
*   [pyate.term_extraction_pipeline](pyate.html#module-pyate.term_extraction_pipeline)
*   [pyate.term_extractor](pyate.html#module-pyate.term_extractor)
*   [pyate.weirdness](pyate.html#module-pyate.weirdness)
```

--------------------------------

### Get General Domain Data

Source: https://github.com/kevinlu1248/pyate/blob/master/html/_modules/pyate/term_extraction.html

Fetches a pandas Series of general domain terms, loading from a zipped CSV file and caching it. Supports specifying language and the number of terms to load.

```python
@staticmethod
def get_general_domain(language: str = None, size: int = None):
    """
    For getting a pandas Series of domains.
    """
    if language is None:
        language = TermExtraction.config["language"]
    if size is None:
        size = TermExtraction.config["DEFAULT_GENERAL_DOMAIN_SIZE"]
    if (language, size) not in TermExtraction.DEFAULT_GENERAL_DOMAINS:
        TermExtraction.DEFAULT_GENERAL_DOMAINS[(language, size)] = pd.read_csv(
            pkg_resources.resource_stream(
                __name__,
                f"default_general_domain.{language}.zip",
            ),
            nrows=size,
        )
    return TermExtraction.DEFAULT_GENERAL_DOMAINS[(language, size)]
```

--------------------------------

### Get spaCy NLP Model

Source: https://github.com/kevinlu1248/pyate/blob/master/html/_modules/pyate/term_extraction.html

Retrieves and caches spaCy NLP models for a given language. Ensures that only one instance of the NLP model is loaded per language to save memory.

```python
@staticmethod
def get_nlp(language: str = None):
    """
    For getting the spaCy nlp.
    """
    if language is None:
        language = TermExtraction.language
    if language not in TermExtraction.nlps:
        TermExtraction.nlps[language] = spacy.load(
            TermExtraction.config["spacy_model"],
            parser=False,
            entity=False,
        )
    return TermExtraction.nlps[language]
```

--------------------------------

### POST /configure

Source: https://github.com/kevinlu1248/pyate/blob/master/html/pyate.html

Updates the configuration settings for the term extraction process.

```APIDOC
## POST /configure

### Description
Updates the configuration settings for the TermExtraction class, including spaCy model, language, and word length constraints.

### Method
POST

### Parameters
#### Request Body
- **new_settings** (Dict[str, Any]) - Required - A dictionary containing settings such as 'spacy_model', 'language', 'MAX_WORD_LENGTH', 'DEFAULT_GENERAL_DOMAIN_SIZE', and 'dtype'.
```

--------------------------------

### Helper function to get subsequences

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_modules/pyate/combo_basic.html

Generates all possible contiguous subsequences of words from a given string, excluding the entire string itself. Useful for analyzing term occurrences within a larger text.

```python
import math
from typing import List, Mapping, Sequence

import pandas as pd
import numpy as np
from tqdm import tqdm

from .term_extraction import TermExtraction, add_term_extraction_method, Corpus

def helper_get_subsequences(s: str) -> List[str]:
    sequence = s.split()
    if len(sequence) <= 2:
        return []
    answer = []
    for left in range(len(sequence) + 1):
        for right in range(left + 1, len(sequence) + 1):
            if left == 0 and right == len(sequence):
                continue
            answer.append(" ".join(sequence[left:right]))
    return answer
```

--------------------------------

### Setting Language for Term Extraction

Source: https://context7.com/kevinlu1248/pyate/llms.txt

Configure PyATE to use different languages by setting the spaCy model and language code. Supports Italian, German, and Spanish, with a reset to English. Requires corresponding spaCy models to be installed.

```python
from pyate import TermExtraction, combo_basic

# Switch to Italian
TermExtraction.set_language("it", "it_core_news_sm")

italian_corpus = """L'intelligenza artificiale sta trasformando il settore sanitario.
L'apprendimento automatico consente la diagnosi precoce delle malattie.
Le reti neurali analizzano immagini mediche per rilevare anomalie."""

italian_terms = combo_basic(italian_corpus)
print("Italian terms:")
print(italian_terms.sort_values(ascending=False))

# Switch to German
TermExtraction.set_language("de", "de_core_news_sm")

german_corpus = """Maschinelles Lernen verbessert die Datenanalyse.
Neuronale Netze erkennen komplexe Muster in Daten.
Künstliche Intelligenz automatisiert repetitive Aufgaben."""

german_terms = combo_basic(german_corpus)
print("\nGerman terms:")
print(german_terms.sort_values(ascending=False))

# Switch to Spanish
TermExtraction.set_language("es", "es_core_news_sm")

spanish_corpus = """El aprendizaje automático revoluciona la medicina.
Las redes neuronales procesan grandes volúmenes de datos.
La inteligencia artificial mejora el diagnóstico médico."""

spanish_terms = combo_basic(spanish_corpus)
print("\nSpanish terms:")
print(spanish_terms.sort_values(ascending=False))

# Reset to English
TermExtraction.set_language("en", "en_core_web_sm")
```

--------------------------------

### PyATE Library Index

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/genindex.html

This section lists the main modules and classes available in the PyATE library, along with their associated functions and attributes.

```APIDOC
## PyATE Library Components

### Description
This documentation provides an index of the PyATE library, detailing its modules, classes, functions, and attributes.

### Modules
- **pyate**
- **pyate.basic**
- **pyate.combo_basic**
- **pyate.cvalues**
- **pyate.term_extraction**
- **pyate.term_extraction_pipeline**
- **pyate.term_extractor**
- **pyate.weirdness

### Classes
- **TermExtraction** (in module pyate.term_extraction)
- **TermExtractionPipeline** (in module pyate.term_extraction_pipeline)

### Functions and Methods

#### pyate.term_extraction
- **add_term_extraction_method()**
- **TermExtraction.adj**
- **TermExtraction.basic()**
- **TermExtraction.clear_resouces()** (static method)
- **TermExtraction.combo_basic()**
- **TermExtraction.config**
- **TermExtraction.configure()** (static method)
- **TermExtraction.count_terms_from_document()**
- **TermExtraction.count_terms_from_documents()**
- **TermExtraction.cvalues()**
- **TermExtraction.DEFAULT_GENERAL_DOMAINS**
- **TermExtraction.get_general_domain()** (static method)
- **TermExtraction.get_nlp()** (static method)
- **TermExtraction.nlps**
- **TermExtraction.noun**
- **TermExtraction.patterns**
- **TermExtraction.prep**
- **TermExtraction.set_language()** (static method)
- **TermExtraction.term_extractor()**
- **TermExtraction.trie

#### pyate.basic
- **basic()**

#### pyate.combo_basic
- **combo_basic()**
- **helper_get_subsequences()**

#### pyate.cvalues
- **cvalues()**
- **helper_get_subsequences()**

#### pyate.term_extractor
- **term_extractor()**

#### pyate.weirdness
- **weirdness()**

```

--------------------------------

### Set Language for Term Extraction

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_modules/pyate/term_extraction.html

Changes the language used by TermExtraction. A warning is issued if the provided model name does not start with the specified language, indicating potential incompatibility. The default general domain is currently only available in English and Italian.

```python
TermExtraction.set_language("en", "en_core_web_sm")
```

--------------------------------

### Configure PyATE settings

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/_sources/general.rst.txt

Updates global library settings such as the data type for frequency counters.

```python
import numpy as np
TermExtraction.configure({"dtype": np.uint32})
```

--------------------------------

### Configure Term Extraction Settings

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_modules/pyate/term_extraction.html

Updates configuration settings for TermExtraction. This includes options for the spaCy model, language, maximum word length for phrases, default domain size, and data type for counters. Ensure that the configured model name starts with the language code for compatibility.

```python
TermExtraction.configure({
    "spacy_model": "en_core_web_sm",
    "language": "en",
    "MAX_WORD_LENGTH": 6,
    "DEFAULT_GENERAL_DOMAIN_SIZE": 300,
    "dtype": np.int16
})
```

--------------------------------

### Basic Algorithm

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/algorithms.html

The Basic algorithm is a simplified version of ComboBasic, focusing on term length and frequency, with an additional factor for the number of term candidates containing the term.

```APIDOC
## POST /api/basic

### Description
Applies the Basic term extraction algorithm to the provided technical corpus.

### Method
POST

### Endpoint
/api/basic

### Parameters
#### Request Body
- **technical_corpus** (str) - Required - The corpus from which technical terms are to be searched.
- **general_corpus** (pd.Series) - Optional - The corpus to compare against. Defaults to TermExtraction.get_general_domain().
- **general_corpus_size** (int) - Optional - The number of sentences desired for the general corpus. Defaults to 300.
- **verbose** (bool) - Optional - Whether to display a progress bar. Defaults to False.
- **weights** (Sequence[float]) - Optional - Configurable weights for the algorithm's factors. Defaults are [1, 3.5].
- **smoothing** (double) - Optional - Factor added before division to prevent zero division errors. Defaults to 0.01.
- **have_single_word** (bool) - Optional - Determines if single words are considered terms. Defaults to False.
- **technical_counts** (Mapping[str, int]) - Optional - Pre-computed frequency counts for technical terms. If not provided, it will be computed.

### Request Example
```json
{
  "technical_corpus": "I am a term extractor!",
  "general_corpus_size": 500,
  "weights": [1.0, 3.5]
}
```

### Response
#### Success Response (200)
- **terms** (list) - A list of extracted terms.

#### Response Example
```json
{
  "terms": ["term extractor"]
}
```
```

--------------------------------

### Initialize TermExtraction Class

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/docs/_build/html/_modules/pyate/term_extraction.html

Initializes the TermExtraction class with corpus, vocabulary, patterns, and language settings. Supports parallel processing if enabled.

```python
def __init__(
        self,
        corpus: Union[str, Iterable[str]],
        vocab: Sequence[str] = None,
        patterns=patterns,
        do_parallelize: bool = True,
        language="en",
        nlp=None,
        default_domain=None,
        default_domain_size: int = None,
        max_word_length: int = None,
        dtype: np.dtype = None,
    ):
        """
        If corpus is a string, then find vocab sequentially, but if the corpus is an iterator, compute through each of them, performing in parallel if do_parallel is set to true. If there is a vocab list, only search for frequencies from the vocab list, otherwise search using the patterns.
        """
        # TODO: find a way to cache counts for general domain
        self.corpus = corpus
        self.vocab = vocab
        self.patterns = patterns
        self.do_parallelize = do_parallelize
        self.language = language
        self.nlp = nlp
        self.default_domain = default_domain
        self.default_domain_size = default_domain_size
        self.max_word_length = max_word_length
        self.dtype = dtype
        if self.default_domain_size is None:

```

--------------------------------

### pyate.basic.basic

Source: https://github.com/kevinlu1248/pyate/blob/master/html/pyate.html

Extracts terms using the basic algorithm.

```APIDOC
## pyate.basic.basic

### Description
Performs term extraction on a technical corpus using the basic algorithm.

### Parameters
#### Request Body
- **technical_corpus** (str) - Required - The input text corpus.
- **args** (list) - Optional - Additional arguments.
- **kwargs** (dict) - Optional - Additional keyword arguments.
```

--------------------------------

### Basic Algorithm

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/algorithms.html

The Basic algorithm is a simplified version of ComboBasic, focusing on term length and frequency, with an additional factor for the number of term candidates containing the candidate.

```APIDOC
## Basic Algorithm

### Description
This algorithm is equivalent to `combo_basic(technical_corpus, weights=np.array([1, 3.5, 0]), *args, **kwargs)`. It calculates term scores based on length and frequency, with an additional factor for the number of term candidates containing the candidate.

### Method
`pyate.basic`

### Endpoint
N/A (Python function)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **technical_corpus** (str) - Required - The corpus from which technical terms are to be searched.
- ***args** - Additional positional arguments.
- ****kwargs** - Additional keyword arguments.

### Request Example
```python
from pyate import basic
basic("I am a term extractor!")
```

### Response
#### Success Response (200)
Returns a list of extracted terms.

#### Response Example
```json
{
  "terms": ["term extractor"]
}
```
```

--------------------------------

### Combo Basic Algorithm

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_sources/algorithms.rst.txt

The Combo Basic algorithm, an enhanced version of the Basic algorithm.

```APIDOC
## POST /pyate/combo_basic

### Description
Implements the Combo Basic algorithm for term extraction.

### Method
POST

### Endpoint
/pyate/combo_basic

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **technical_corpus** (Union[str, Sequence[str]]) - Required - The corpus from which technical terms are to be searched.
- **general_corpus** (pd.Series) - Optional - The corpus to compare against. Defaults to TermExtraction.get_general_domain().
- **general_corpus_size** (int) - Optional - The number of sentences desired for the general corpus. Defaults to 300.
- **verbose** (bool) - Optional - Whether to display a progress bar. Defaults to False.
- **weights** (dict) - Optional - Defines the weights for various factors in term likelihood calculation. Meaning varies by algorithm.
- **smoothing** (double) - Optional - A factor added to numbers before division to prevent division by zero errors. Defaults to 0.01.
- **have_single_word** (bool) - Optional - Determines if single words should be considered as terms. Defaults to False.
- **technical_counts** (Mapping[str, int]) - Optional - A frequency counter for technical terms. If not provided, it will be computed.
- **threshold** (float) - Optional - The threshold for the C-Value algorithms. Defaults to 0.

### Request Example
```json
{
  "technical_corpus": "A document for combo basic.",
  "weights": {"factor1": 0.7, "factor2": 0.3}
}
```

### Response
#### Success Response (200)
- **terms** (list) - A list of extracted terms.

#### Response Example
```json
{
  "terms": ["combo term 1", "combo term 2"]
}
```
```

--------------------------------

### pyate.combo_basic.helper_get_subsequences

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/docs/_build/html/pyate.html

Helper function to generate subsequences of a given string.

```APIDOC
## pyate.combo_basic.helper_get_subsequences

### Description
Generates all possible contiguous subsequences of a given string.

### Method
Function

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
None
```

--------------------------------

### pyate.basic Module

Source: https://github.com/kevinlu1248/pyate/blob/master/docs/pyate.html

Provides basic term extraction functionality.

```APIDOC
## pyate.basic.basic

### Description
Extracts basic terms from a technical corpus.

### Method
`basic`

### Parameters
- **technical_corpus** (str) - The technical corpus to process.
- **args** - Additional positional arguments.
- **kwargs** - Additional keyword arguments.
```

--------------------------------

### TermExtractionPipeline Initialization

Source: https://github.com/kevinlu1248/pyate/blob/master/html/_modules/pyate/term_extraction_pipeline.html

Initializes the spaCy pipeline component for term extraction.

```APIDOC
## Class: TermExtractionPipeline

### Description
Initializes the PyATE pipeline component for integration with spaCy. It sets up the matcher and registers a custom extension on the spaCy Doc object.

### Parameters
- **nlp** (object) - Required - The spaCy language object.
- **func** (Callable) - Optional - The extraction function to use (defaults to combo_basic).
- **force** (bool) - Optional - Whether to force the extension registration on the Doc object (defaults to True).
- **args** (list) - Optional - Additional positional arguments for the extraction function.
- **kwargs** (dict) - Optional - Additional keyword arguments for the extraction function.
```

--------------------------------

### Implement basic term extraction in PyATE

Source: https://github.com/kevinlu1248/pyate/blob/master/docsrc/_build/html/_modules/pyate/basic.html

Defines the basic term extraction function using predefined weights and demonstrates usage with a sample corpus.

```python
# basic.py

import numpy as np

from .term_extraction import TermExtraction, add_term_extraction_method
from .combo_basic import combo_basic

[docs]../../algorithms.html#pyate.basic.basic@add_term_extraction_method
def basic(technical_corpus: str, *args, **kwargs):
    # TODO: optimize
    weights = np.array([1, 3.5, 0])
    return combo_basic(technical_corpus, weights=weights, *args, **kwargs)

if __name__ == "__main__":
    corpus = "Hello I am a term extractor."
    print(TermExtraction(corpus).basic().sort_values(ascending=False).head(50))
```