TRUNAJOD

https://trunajod20.readthedocs.io/en/latest
Admin
TRUNAJOD is a Python library for text complexity analysis built on spaCy that extracts linguistic...
Tokens:16,711
Snippets:169
Trust Score:9.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### TRUNAJOD Web App Backend Example

Source: https://trunajod20.readthedocs.io/en/latest/index.html

This example demonstrates the backend setup for the TRUNAJOD web application, which assesses text complexity and school level adequacy. It processes .docx files, extracts text, and computes various TRUNAJOD indices using spaCy and pandas for analysis and visualization.

```python
"""Example of TRUNAJOD usage."""
import glob

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import spacy
import textract  # To read .docx files
import TRUNAJOD.givenness
import TRUNAJOD.ttr
from TRUNAJOD import surface_proxies
from TRUNAJOD.syllabizer import Syllabizer

plt.rcParams["figure.figsize"] = (11, 4)
plt.rcParams["figure.dpi"] = 200


nlp = spacy.load("es_core_news_sm", disable=["ner", "textcat"])

features = {
    "lexical_diversity_mltd": [],
    "lexical_density": [],
    "pos_dissimilarity": [],
    "connection_words_ratio": [],
    "grade": [],
}
for filename in glob.glob("corpus/*/*.docx"):
    text = textract.process(filename).decode("utf8")
    doc = nlp(text)
    features["lexical_diversity_mltd"].append(
        TRUNAJOD.ttr.lexical_diversity_mtld(doc)
    )
    features["lexical_density"].append(surface_proxies.lexical_density(doc))
    features["pos_dissimilarity"].append(
        surface_proxies.pos_dissimilarity(doc)
    )
    features["connection_words_ratio"].append(
        surface_proxies.connection_words_ratio(doc)
    )

    # In our case corpus was organized as:
    # corpus/5B/5_2_55.docx where the folder that
    # contained the doc, contained the school level, in
    # this example 5th grade
    features["grade"].append(filename.split("/")[1][0])

df = pd.DataFrame(features)


fig, axes = plt.subplots(2, 2)

sns.boxplot(x="grade", y="lexical_diversity_mltd", data=df, ax=axes[0, 0])
sns.boxplot(x="grade", y="lexical_density", data=df, ax=axes[0, 1])
sns.boxplot(x="grade", y="pos_dissimilarity", data=df, ax=axes[1, 0])
sns.boxplot(x="grade", y="connection_words_ratio", data=df, ax=axes[1, 1])
```

--------------------------------

### Load TRUNAJOD and spaCy Models for Text Analysis

Source: https://trunajod20.readthedocs.io/en/latest/index.html

This snippet demonstrates how to load pre-built TRUNAJOD models and a spaCy language model. It includes a custom ModelLoader class for TRUNAJOD models and initializes a spaCy pipeline, disabling specific components for efficiency. Ensure the specified spaCy model is installed.

```python
from TRUNAJOD import surface_proxies
from TRUNAJOD.entity_grid import EntityGrid
from TRUNAJOD.lexico_semantic_norms import LexicoSemanticNorm
import pickle
import spacy
import tarfile


class ModelLoader(object):
    """Class to load model."""
    def __init__(self, model_file):
        tar = tarfile.open(model_file, "r:gz")
        self.crea_frequency = {}
        self.infinitive_map = {}
        self.lemmatizer = {}
        self.spanish_lexicosemantic_norms = {}
        self.stopwords = {}
        self.wordnet_noun_synsets = {}
        self.wordnet_verb_synsets = {}

        for member in tar.getmembers():
            f = tar.extractfile(member)
            if "crea_frequency" in member.name:
                self.crea_frequency = pickle.loads(f.read())
            if "infinitive_map" in member.name:
                self.infinitive_map = pickle.loads(f.read())
            if "lemmatizer" in member.name:
                self.lemmatizer = pickle.loads(f.read())
            if "spanish_lexicosemantic_norms" in member.name:
                self.spanish_lexicosemantic_norms = pickle.loads(f.read())
            if "stopwords" in member.name:
                self.stopwords = pickle.loads(f.read())
            if "wordnet_noun_synsets" in member.name:
                self.wordnet_noun_synsets = pickle.loads(f.read())
            if "wordnet_verb_synsets" in member.name:
                self.wordnet_verb_synsets = pickle.loads(f.read())


# Load TRUNAJOD models
model = ModelLoader("trunajod_models_v0.1.tar.gz")

# Load spaCy model
lp = spacy.load("es_core_news_sm", disable=["ner", "textcat"])

example_text = (
    "El espectáculo del cielo nocturno cautiva la mirada y suscita preguntas"
    "sobre el universo, su origen y su funcionamiento. No es sorprendente que "
    "todas las civilizaciones y culturas hayan formado sus propias "
    "cosmologías. Unas relatan, por ejemplo, que el universo ha"
    "sido siempre tal como es, con ciclos que inmutablemente se repiten; "
    "otras explican que este universo ha tenido un principio, "
    "que ha aparecido por obra creadora de una divinidad."
)

doc = nlp(example_text)

# Lexico-semantic norms
lexico_semantic_norms = LexicoSemanticNorm(
    doc,
    model.spanish_lexicosemantic_norms,
    model.lemmatizer
)

# Frequency index
freq_index = surface_proxies.frequency_index(doc, model.crea_frequency)

```

--------------------------------

### infinitve

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Get the infinitive form of a conjugated verb using a provided mapping. Models are available for Spanish.

```APIDOC
## infinitve(conjugate, infinitive_map)

### Description
Get infinitive form of a conjugated verb. Given a mapping of conjugate to infinitive, this function computes the infinitive form of a conjugate verb. We provide models available for downloading, so you do not have to worry about the `infinitive_map`. Regretfully we only provide models for Spanish texts.

### Parameters
* **conjugate** (string) – Verb to be processed
* **infinitve_map** – Lexicon containing maps from conjugate to infinitive.

### Returns
Infinitive form of the verb, None if not found

### Return type
string
```

--------------------------------

### pos_distribution

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Get the Part of Speech (POS) distribution from a processed text as a dictionary.

```APIDOC
## pos_distribution(doc)

### Description
Get POS distribution from a processed text. Let us suppose that a given sentence has the following pos tags: `[NOUN, VERB, ADJ, VERB, ADJ]`. The PoS distribution would be
  * `NOUN: 1`
  * `VERB: 2`
  * `ADJ: 2`

This function returns this distrubution as a dict.

### Parameters
**doc** (Spacy Doc) – Processed text

### Returns
POS distribution as a dict key POS, value Count

### Return type
dict
```

--------------------------------

### Calculate TRUNAJOD Metrics

Source: https://trunajod20.readthedocs.io/en/latest/index.html

Calculates various linguistic metrics for a given document using the TRUNAJOD library. Requires a spaCy document and a model's infinitive map.

```python
clause_count = surface_proxies.clause_count(doc, model.infinitive_map)

# Compute Entity Grid
egid = EntityGrid(doc)

print("Concreteness: {}".format(lexico_semantic_norms.get_concreteness()))
print("Frequency Index: {}".format(freq_index))
print("Clause count: {}".format(clause_count))
print("Entity grid:")
print(egrid.get_egrid())
```

--------------------------------

### TRUNAJOD.surface_proxies.connection_words_ratio

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Calculates the ratio of connection words to the total number of words in a Spanish text. Considers specific lemmas like 'y', 'o', 'no', 'si'.

```APIDOC
## TRUNAJOD.surface_proxies.connection_words_ratio

### Description
Get ratio of connecting words over total words of text. This function computes the ratio of connective words over the total number of words. This implementation is only supported in Spanish and we consider the following lemmas: `y`, `o`, `no`, `si`.

### Parameters
**doc** (_Spacy Doc_) – Tokenized text

### Returns
Connection word ratio

### Return type
float
```

--------------------------------

### noun_phrase_density

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Compute noun phrase (NP) density heuristically. It calculates the ratio of NPs to the noun count.

```APIDOC
## noun_phrase_density(doc)

### Description
Compute NP density. To compute NP density we do it heuristically. We might improve it in the future by using some NP-chunking strategy. For counting noun phrases, we check that for a node in the parse tree, its head is a Noun. Then, we check if either of the following conditions is met:
  * The token is the article `del` or `al`
  * The token dependency is not `cc`, `case` or `cop`, and the token is not a punctuation and the token is not the `ROOT`

Then we compute the ratio between # of NP / Noun count.

### Parameters
**doc** (Spacy Doc) – Tokenized text.

### Returns
NP density

### Return type
float
```

--------------------------------

### EntityGrid Class

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/entity_grid.html

The EntityGrid class creates an entity grid from a spaCy processed document. It supports 2-transitions entity grids and relies on the spaCy module.

```APIDOC
## Class `TRUNAJOD.entity_grid.EntityGrid`

### Description
Entity grid class. Creates an entity grid from a doc, which is the output of applying spacy.nlp(text) to a text. This class depends on the spaCy module and only supports 2-transitions entity grids.

### Methods

- `get_egrid()`: Returns the obtained entity grid for debugging purposes.
- `get_nn_transitions()`: Returns the ratio of NN transitions.
- `get_no_transitions()`: Returns the ratio of N-O transitions.
- `get_ns_transitions()`: Returns the ratio of N-S transitions.
- `get_nx_transitions()`: Returns the ratio of NX transitions.
- `get_on_transitions()`: Returns the ratio of ON transitions.
- `get_oo_transitions()`: Returns the ratio of OO transitions.
- `get_os_transitions()`: Returns the ratio of OS transitions.
- `get_ox_transitions()`: Returns the ratio of OX transitions.
- `get_sentence_count()`: Returns the number of sentences obtained during processing.
- `get_sn_transitions()`: Returns the ratio of SN transitions.
- `get_so_transitions()`: Returns the ratio of SO transitions.
- `get_ss_transitions()`: Returns the ratio of SS transitions.
- `get_sx_transitions()`: Returns the ratio of SX transitions.
- `get_xn_transitions()`: Returns the ratio of X- transitions.
- `get_xo_transitions()`: Returns the ratio of XO transitions.
- `get_xs_transitions()`: Returns the ratio of XS transitions.
- `get_xx_transitions()`: Returns the ratio of XX transitions.

### Returns
- `get_egrid()`: dict - Entity grid represented as a dict.
- `get_nn_transitions()`: float - Ratio of transitions.
- `get_no_transitions()`: float - Ratio of transitions.
- `get_ns_transitions()`: float - Ratio of transitions.
- `get_nx_transitions()`: float - Ratio of transitions.
- `get_on_transitions()`: float - Ratio of transitions.
- `get_oo_transitions()`: float - Ratio of transitions.
- `get_os_transitions()`: float - Ratio of transitions.
- `get_ox_transitions()`: float - Ratio of transitions.
- `get_sentence_count()`: int - Number of sentences.
- `get_sn_transitions()`: float - Ratio of transitions.
- `get_so_transitions()`: float - Ratio of transitions.
- `get_ss_transitions()`: float - Ratio of transitions.
- `get_sx_transitions()`: float - Ratio of transitions.
- `get_xn_transitions()`: float - Ratio of transitions.
- `get_xo_transitions()`: float - Ratio of transitions.
- `get_xs_transitions()`: float - Ratio of transitions.
- `get_xx_transitions()`: float - Ratio of transitions.
```

--------------------------------

### LexicoSemanticNorm Class

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/lexico_semantic_norms.html

Initialize a LexicoSemanticNorm calculator for a given document and lexico-semantic norm dictionary. It computes average values over tokens for specified lexico-semantic variables.

```APIDOC
## class TRUNAJOD.lexico_semantic_norms.LexicoSemanticNorm(doc, lexico_semantic_norm_dict, lemmatizer=None)

### Description
Create a lexico semantic norm calculator for text. This requires a lexico semantic norm dict, with key-value pairs specified as `word -> {"arousal", "concreteness", "context_availability", "familiarity", "imageability", "valence"}`. Average over number of tokens will be computed. The values are obtained from [GFerreF16].

### Parameters
* **doc** (_Spacy Doc_) - The input document to process.
* **lexico_semantic_norm_dict** (_dict_) - A dictionary mapping words to their lexico-semantic norm values.
* **lemmatizer** (_object, optional_) - A lemmatizer object. Defaults to None.
```

--------------------------------

### get_synsets

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/semantic_measures.html

Retrieves the synonym set for a given word lemma. It requires a pre-defined synset dictionary and returns a set containing the lemma if it's not found.

```APIDOC
## get_synsets

### Description
Return synonym set given a word lemma. The function requires that the synset_dict is passed into it. In our case we provide downloadable models from MCR (Multilingual-Central-Repository). [GALR12]. If the lemma is not found in the synset_dict, then this function returns a set with the lemma in it.

### Parameters
#### Path Parameters
- **lemma** (string) - Required - Lemma to be look-up into the synset
- **synset_dict** (Python dict) - Required - key-value pairs, lemma to synset

### Returns
The set of synonyms of a given lemma

### Return Type
Python set of strings
```

--------------------------------

### get_conc_imag_familiarity

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/lexico_semantic_norms.html

Computes Concreteness, Imageability, and Familiarity lexico-semantic variables, averaged over sentences.

```APIDOC
## TRUNAJOD.lexico_semantic_norms.get_conc_imag_familiarity(doc)

### Description
Get lexico-semantic variables. Computes three lexico-semantic variables: Concreteness, Imageability and Familiarity. The values are obtained from the EsPal dictionary (Spanish) and average of each metric is computed over sentences. To get each metric, the best practice is using LSNorm Enum defined in lexicosemantic_norms_espal module. The enums are CONCRETENESS, IMAGEABILITY and FAMILIARITY. This implementation uses values of the lexico-semantic norms from [DPSebastianGalles+13].

### Parameters
* **doc** (_Spacy Doc_) – Tokenized text

### Returns
* **Concreteness imageability and familiarity averaged over sentences** (List of float)
```

--------------------------------

### verb_noun_ratio

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Calculates and returns the ratio of verbs to nouns within a given processed text (Spacy Doc).

```APIDOC
## verb_noun_ratio

### Description
Compute Verb/Noun ratio.

### Parameters
* **doc** (_Spacy Doc_) – Processed text

### Returns
Verb Noun ratio

### Return type
float
```

--------------------------------

### syllable_word_ratio

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Computes and returns the average syllable-to-word ratio for a given text (Spacy Doc). The ratio is calculated as the total number of syllables divided by the total number of words.

```APIDOC
## syllable_word_ratio

### Description
Return average syllable word ratio. It is computed as # Syllables / # of words.

### Parameters
* **doc** (_Spacy Doc_) – Text to be processed.

### Returns
syllable word ratio

### Return type
float
```

--------------------------------

### negation_density

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Compute negation density, defined as the ratio of negation words to the total word count.

```APIDOC
## negation_density(doc)

### Description
Compute negation density. This is defined as the ratio between number of occurrences of `TRUNAJOD.surface_proxies.NEGATION_WORDS` in the text over the total word count.

### Parameters
**doc** (Spacy Doc) – Tokenized text

### Returns
Negation density

### Return type
float
```

--------------------------------

### pronoun_noun_ratio

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/givenness.html

Computes the pronoun-noun ratio of a given SpaCy document. This metric approximates text complexity by calculating the ratio between third-person pronouns and the total number of nouns in the text.

```APIDOC
## pronoun_noun_ratio

### Description
Compute Pronoun Noun ratio. This is an approximation of text complexity/readability, since pronouns are co-references to a proper noun or a noun. This is computed as the taking the ratio between third person pronouns and total nouns.

### Parameters
#### Path Parameters
- **doc** (spacy.tokens.doc.Doc) - Required - Text to be processed.

### Returns
- **float** - pronoun-noun ratio
```

--------------------------------

### TRUNAJOD.surface_proxies.average_sentence_length

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Computes the average sentence length of a text, calculated as the ratio of the total number of words to the total number of sentences.

```APIDOC
## TRUNAJOD.surface_proxies.average_sentence_length

### Description
Return average sentence length. This measurement is computed as the ratio of: # of words / # of sentences.

### Parameters
**doc** (_Spacy Doc_) – Text to be processed.

### Returns
average sentence length

### Return type
float
```

--------------------------------

### get_familiarity

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/lexico_semantic_norms.html

Retrieves the average familiarity value for the processed text.

```APIDOC
## get_familiarity()

### Description
Get familiarity.

### Returns
* **Average familiarity** (float) - The computed average familiarity value.
```

--------------------------------

### pos_ratio

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Compute the ratio of specified Part of Speech types over the total number of words in the text. Supports regular expressions for combined POS types.

```APIDOC
## pos_ratio(doc, pos_types)

### Description
Compute POS ratio given desired type of ratio. The `pos_types` might be a regular expression if a composed ratio is needed. An example of usage would be `pos_ratio(doc, "VERB|AUX")`.

### Parameters
* **doc** (Spacy Doc) – Spacy processed text
* **pos_types** (string) – POS to get the ratio

### Returns
Ratio over number of words

### Return type
float
```

--------------------------------

### TRUNAJOD.surface_proxies.average_clause_length

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Calculates the average clause length of a text using a heuristic approach. It's computed as the ratio of the number of words to the number of clauses.

```APIDOC
## TRUNAJOD.surface_proxies.average_clause_length

### Description
Return average clause length (heuristic). This measurement is computed as the ratio of # of words / # of clauses. To count clauses we do it heuristically, and you can refer to `TRUNAJOD.surface_proxies.clause_count()` for more details.

### Parameters
* **doc** (_Spacy Doc_) – Text to be processed
* **infinitve_map** – Lexicon containing maps from conjugate to infinitive.

### Returns
Average clause length

### Return type
float
```

--------------------------------

### dependency_mapping Function

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/entity_grid.html

Maps a dependency tag to an entity grid tag using the notation provided in [BL08].

```APIDOC
## Function `TRUNAJOD.entity_grid.dependency_mapping`

### Description
Map dependency tag to entity grid tag. Considers the notation provided in [BL08].

| EGrid Tag | Dependency Tag |
|---|---|
| S | nsub, csubj, csubjpass, dsubjpass |
| O | iobj, obj, pobj, dobj |
| X | For any other dependency tag |

### Parameters
- **dep** (string) - Required - Dependency tag.

### Returns
- string - EGrid tag.
```

--------------------------------

### TRUNAJOD.utils.read_text

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Reads a UTF-8 encoded text file and returns its content as a string. Recommended for small text files that fit into RAM.

```APIDOC
## TRUNAJOD.utils.read_text

### Description
Read a `utf-8` encoded text file and returns the text as `string`. This is just a utily function, that is not recommended to use if the text file does not fit your available RAM. Mostly used for small text files.

### Parameters
#### Parameters
- **filename** (string) - File from which is the text to be read

### Returns
Text in the file

### Return Type
string
```

--------------------------------

### syntactic_similarity

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Computes the average syntactic similarity between sentences in a processed text (Spacy Doc). It calculates similarity for each pair of nodes and averages the results.

```APIDOC
## syntactic_similarity

### Description
Compute average syntactic similarity between sentences. For each pair of sentences, compute the similarity between each pair of nodes, using `TRUNAJOD.surface_proxies.node_similarity()` Then, the result is averaged over the `N - 1` pair of sentences.

### Parameters
* **doc** (_Spacy Doc_) – Processed text

### Returns
Average syntactic similarity over sentences.

### Return type
float
```

--------------------------------

### TRUNAJOD.surface_proxies.fix_parse_tree

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Corrects the spaCy parse tree for Spanish texts by addressing issues with periphrasis, gerunds, infinitives, and past tense verbs.

```APIDOC
## TRUNAJOD.surface_proxies.fix_parse_tree

### Description
Fix SPACY parse tree. We found that for Spanish texts, spaCy tags do not deal appropiately with periphrasis and other lingüistic cues. This function address this shortcome by modifying the parse tree computed by spaCy adding periphrasis for Gerunds, Infinitive and Past tense verbs.

### Parameters
* **doc** (_Spacy Doc_) – Processed text
* **infinitve_map** – Lexicon containing maps from conjugate to infinitive.

### Returns
Fixed Doc

### Return type
Spacy Doc
```

--------------------------------

### TRUNAJOD.utils.get_stopwords

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Reads a list of stopwords from a specified file. Assumes words are separated by newlines and uses UTF-8 encoding.

```APIDOC
## TRUNAJOD.utils.get_stopwords

### Description
Read stopword list from file. Assumes that the list is defined as a newline separated words. It is a utility in case you’d like to provide your own stopwords list. Assumes encoding `utf8`.

### Parameters
#### Path Parameters
- **filename** (string) – Name of the file containing stopword list.

### Returns
- List of stopwords (set)
```

--------------------------------

### TRUNAJOD.ttr.lexical_diversity_mtld

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/ttr.html

Computes MTLD (Moving-Average Type-Token Ratio) lexical diversity in a bi-directional fashion. It uses a specified model and a threshold for TTR mean computation.

```APIDOC
## TRUNAJOD.ttr.lexical_diversity_mtld

### Description
Compute MTLD lexical diversity in a bi-directional fashion.

### Parameters
#### Parameters
- **doc** (_NLP Doc_) – Processed text
- **model_name** (_str_) – Determines which model is used (spacy or stanza), defaults to 'spacy'
- **ttr_segment** (_float_) – Threshold for TTR mean computation, defaults to 0.72

### Returns
- Bi-directional lexical diversity MTLD (float)
```

--------------------------------

### get_context_availability

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/lexico_semantic_norms.html

Retrieves the average context availability value for the processed text.

```APIDOC
## get_context_availability()

### Description
Get context_availability.

### Returns
* **Average context_availability** (float) - The computed average context availability value.
```

--------------------------------

### get_imageability

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/lexico_semantic_norms.html

Retrieves the average imageability value for the processed text.

```APIDOC
## get_imageability()

### Description
Get imageability.

### Returns
* **Average imageability** (float) - The computed average imageability value.
```

--------------------------------

### get_concreteness

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/lexico_semantic_norms.html

Retrieves the average concreteness value for the processed text.

```APIDOC
## get_concreteness()

### Description
Get concreteness.

### Returns
* **Average concreteness** (float) - The computed average concreteness value.
```

--------------------------------

### TRUNAJOD.surface_proxies.first_second_person_density

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Computes the density of first and second person pronouns within a processed text.

```APIDOC
## TRUNAJOD.surface_proxies.first_second_person_density

### Description
Compute density of first|second person.

### Parameters
**doc** (_Spacy Doc_) – Processed text

### Returns
Density 1,2 person

### Return type
float
```

--------------------------------

### TRUNAJOD.utils.process_text

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Processes text by tokenizing sentences using a provided tokenizer.

```APIDOC
## TRUNAJOD.utils.process_text

### Description
Process text by tokenizing sentences given a tokenizer.

### Parameters
#### Parameters
- **text** (string) - Text to be processed
- **sent_tokenize** (Python callable that returns list of strings) - Tokenizer

### Returns
Tokenized sentences

### Return Type
List of strings
```

--------------------------------

### node_similarity

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Compute node similarity recursively based on common children POS. This is an auxiliary function for syntactic_similarity and is mainly for debugging.

```APIDOC
## node_similarity(node1, node2, is_central_node=False)

### Description
Compute node similarity recursively, based on common children POS. This function is called inside `TRUNAJOD.surface_proxies.syntactic_similarity()` so is an auxiliary function. In the common use case, is unlikely you will need to call this function directly, but we provide it for debugging purposes.

### Parameters
* **node1** (Spacy Token) – Node of the parse tree.
* **node2** (Spacy Token) – Node of the parse tree
* **is_central_node** (bool, optional) – Whether is the central node, defaults to False

### Returns
Total childs in common between node1 and node2.

### Return type
int
```

--------------------------------

### weighting_syntactic_role Function

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/entity_grid.html

Returns a weight for a given entity's grammatical role based on the heuristic from [GS13].

```APIDOC
## Function `TRUNAJOD.entity_grid.weighting_syntactic_role`

### Description
Return weight given an entity grammatical role. This uses the heuristic from [GS13].

| EGrid Tag | Weight |
|---|---|
| S | 3 |
| O | 2 |
| X | 1 |
| dash | 0 |

### Parameters
- **entity_role** (string) - Required - Entity grammatical role (S, O, X, -).

### Returns
- int - Role weight.
```

--------------------------------

### TRUNAJOD.utils.get_sentences_lemmas

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Extracts various lemma types (noun, verb, content, etc.) from sentences in a list of Spacy Docs. It utilizes internal functions to measure different lemma types.

```APIDOC
## TRUNAJOD.utils.get_sentences_lemmas

### Description
Get lemmas from sentences. Get different types of lemma measurements, such as noun lemmas, verb lemmas, content lemmas. It calls `TRUNAJOD.utils.get_token_lemmas()` internally to extract different lemma types for each sentence. This function extract the following lemmas:
  * Noun lemmas
  * Verb lemmas
  * Function lemmas (provided as `stopwords`)
  * Content lemmas (anything that is not in `stopwords`)
  * Adjective lemmas
  * Adverb lemmas
  * Proper pronoun lemmas

### Parameters
#### Path Parameters
- **docs** (List of Spacy Doc (`Doc.sents`)) – List of sentences to be processed.
- **lemma_dict** (dict) – Lemmatizer dictionary
- **stopwords** (list, optional) – List of stopwords (function words), defaults to []

### Returns
- List of lemmas from text (List of Lists of str)
```

--------------------------------

### TRUNAJOD.utils.lemmatize

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Lemmatizes a given word using a provided dictionary mapping words to their lemmas. If a word is not found, it returns the word itself.

```APIDOC
## TRUNAJOD.utils.lemmatize

### Description
Lemmatize a word. Lemmatizes a word using a lemmatizer which is represented as a dict that has (word, lemma) as (key, value) pair. An example of a lemma list can be found in https://github.com/michmech/lemmatization-lists. If the word is not found in the dictionary, the lemma returned will be the word.

### Parameters
#### Path Parameters
- **lemma_dict** (Python dict) – A dict (word, lemma)
- **word** (string) – The word to be lemmatized

### Returns
- Lemmatized word (string)
```

--------------------------------

### noun_count

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Count the number of nouns (NOUN or PROPN) in the provided text.

```APIDOC
## noun_count(doc)

### Description
Count nouns in the text. Count all tokens which Part of Speech tag is either `NOUN` or `PROPN`.

### Parameters
**doc** (Spacy Doc) – Text to be processed

### Returns
Noun count

### Return type
int
```

--------------------------------

### TRUNAJOD.utils.get_token_lemmas

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Extracts various lemma types from a Spacy Doc (sentence). It identifies noun, verb, function, content, adjective, adverb, and proper pronoun lemmas.

```APIDOC
## TRUNAJOD.utils.get_token_lemmas

### Description
Return lemmas from a sentence. From a sentence, extracts the following lemmas:
  * Noun lemmas
  * Verb lemmas
  * Function lemmas (provided as `stopwords`)
  * Content lemmas (anything that is not in `stopwords`)
  * Adjective lemmas
  * Adverb lemmas
  * Proper pronoun lemmas

### Parameters
#### Path Parameters
- **doc** (Spacy Doc) – Doc containing tokens from text
- **lemma_dict** (Dict) – Lemmatizer key-value pairs
- **stopwords** (set/list, optional) – list of stopwords, defaults to []

### Returns
- All lemmas for noun, verb, etc. (tuple of lists)
```

--------------------------------

### TRUNAJOD.surface_proxies.average_word_length

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Calculates the average word length in a given text, defined as the ratio of the total number of characters to the total number of words.

```APIDOC
## TRUNAJOD.surface_proxies.average_word_length

### Description
Return average word length. Computed as the ratio of: # number of chars / # of words

### Parameters
**doc** (_Spacy Doc_) – Text to be processed.

### Returns
Average word length

### Return type
float
```

--------------------------------

### Emotions Class

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/emotions.html

The Emotions class is used to compute emotions from a given Spanish text. It takes a spaCy Doc object and an optional lemmatizer as input. The class then provides methods to retrieve the average score for each detected emotion.

```APIDOC
## class `Emotions`

### Description
Compute emotions from SPANISH EMOTION LEXICON [RSG14]. Computes emotions based on Probability Factor of Affective use (PFA) and averaging over the total analized words from the text (that are found in the lexicon).

### Parameters
#### Path Parameters
- **doc** (spacy.tokens.doc.Doc) - Required - The input document to analyze.
- **lemmatizer** (Optional[Dict[str, str]]) - Optional - A dictionary for lemmatization.

### Methods

#### `get_alegria()`

##### Description
Get the average score for 'alegría'.

##### Returns
- **float**: Average alegría over the number of tokens.

#### `get_enojo()`

##### Description
Get the average score for 'enojo'.

##### Returns
- **float**: Average enojo over the number of tokens.

#### `get_miedo()`

##### Description
Get the average score for 'miedo'.

##### Returns
- **float**: Average miedo over the number of tokens.

#### `get_repulsion()`

##### Description
Get the average score for 'repulsión'.

##### Returns
- **float**: Average repulsión over the number of tokens.

#### `get_sorpresa()`

##### Description
Get the average score for 'sorpresa'.

##### Returns
- **float**: Average sorpresa over the number of tokens.

#### `get_tristeza()`

##### Description
Get the average score for 'tristeza'.

##### Returns
- **float**: Average tristeza over the number of tokens.
```

--------------------------------

### pos_dissimilarity

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Measure Part of Speech dissimilarity between sentences by comparing their POS tag distributions.

```APIDOC
## pos_dissimilarity(doc)

### Description
Measure Part of Speech dissimilarity over sentences. The dissimilarity of POS between two sentences is the difference between POS distribution over the total population of POS tags. It is computed as follows:
  * For each sentence, PoS tag distribution is computed.
  * For each tag in either of the two sentences, we compute the difference in distributions (absolute value)
  * This difference is divided by the total population of the sentences

This is done for each pair of sentences (`N - 1` sentences) and the results are averaged (again, over `N - 1`)

### Parameters
**doc** (Spacy Doc) – Processed text

### Returns
Part of Speech dissimilarity

### Return type
float
```

--------------------------------

### lexical_density

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Compute lexical density, defined as the ratio of specific Part of Speech tags (VERB, AUX, ADJ, NOUN, PROPN, ADV) over the total number of words.

```APIDOC
## lexical_density(doc)

### Description
Compute lexical density. The lexical density is defined as the Part of Speech ratio of the following tags: `VERB`, `AUX`, `ADJ`, `NOUN`, `PROPN` and `ADV` over the total number of words.

### Parameters
**doc** (Spacy Doc) – Tokenized text

### Returns
Lexical density

### Return type
Float
```

--------------------------------

### subordination

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Calculates the subordination index of a text, defined as the ratio between the number of clauses and the number of sentences. A heuristic is used to compute the number of clauses.

```APIDOC
## subordination

### Description
Return subordination, defined as the clause density. The subordination is defined as the ratio between # of clauses and the # of sentences. To compute number of clauses, a heuristic is used.

### Parameters
* **doc** (_Spacy Doc_) – Text to be processed.
* **infinitve_map** – Lexicon containing maps from conjugate to infinitive.

### Returns
Subordination index

### Return type
float
```

--------------------------------

### TRUNAJOD.surface_proxies.clause_count

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Provides a heuristic count of clauses in a text. This function is decorated by fix_parse_tree to improve accuracy.

```APIDOC
## TRUNAJOD.surface_proxies.clause_count

### Description
Return clause count (heuristic). This function is decorated by the `TRUNAJOD:surface_proxies.fix_parse_tree()` function, in order to heuristically count clauses.

### Parameters
* **doc** (_Spacy Doc_) – Text to be processed.
* **infinitve_map** – Lexicon containing maps from conjugate to infinitive.

### Returns
Clause count

### Return type
int
```

--------------------------------

### words_before_root

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Computes the average word count of words appearing before the root in sentences. If the root is not a verb, the verb in the highest node of the parse tree is considered the root. An optional `max_depth` parameter can be provided.

```APIDOC
## words_before_root

### Description
Return average word count of words before root. For each sentence, word count before root is computed in the case that the root is a verb. Otherwise, the root is considered to be the verb in the highest node in the parse tree.

### Parameters
* **doc** (_Spacy Doc_) – Text to be processed.
* **max_depth** (int, optional) – Defaults to 4.

### Returns
Average words before root

### Return type
float
```

--------------------------------

### TRUNAJOD.utils.is_pronoun

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Checks if a given token's part-of-speech tag is 'PRON'.

```APIDOC
## TRUNAJOD.utils.is_pronoun

### Description
Return `True` if `pos_tag` is `PRON`, False otherwise.

### Parameters
#### Path Parameters
- **pos_tag** (string) – Part of Speech tag

### Returns
- True if POS is pronoun (boolean)
```

--------------------------------

### find_matches

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/discourse_markers.html

Counts the occurrences of a given list of words (discourse markers) within a text, ignoring case.

```APIDOC
## TRUNAJOD.discourse_markers.find_matches

### Description
Return matches of words in list in a target text. Given a text and a list of possible matches (in this module, discourse markers list), returns the number of matches found in text. This ignores case.

### Hint
For non-Spanish users You could use this function with your custom list of discourse markers in case you need to compute this metric. In that case, the way to call the funcion would be: `find_matches(YOUR_TEXT, ["dm1", "dm2", etc])`

### Parameters
#### Path Parameters
- **text** (string) - Required - Text to be processed
- **list** (Python list of strings) - Required - list of discourse markers

### Returns
Number of ocurrences

### Return type
int
```

--------------------------------

### avg_w2v_semantic_similarity

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/semantic_measures.html

Computes the average semantic similarity between adjacent sentences using a word2vec model based on SPACY implementation and the [FKL98] approach for text coherence.

```APIDOC
## avg_w2v_semantic_similarity

### Description
Compute average semantic similarity between adjacent sentences. This is using word2vec [MCC+13] model based on SPACY implementation. The semantic similarity is based on [FKL98] approach to compute text coherence.

### Parameters
#### Path Parameters
- **docs** (Doc Generator) - Required - Docs generator provided by SPACY API
- **N** (int) - Required - Number of sentences

### Returns
Average sentence similarity (cosine)

### Return Type
float
```

--------------------------------

### overlap

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/semantic_measures.html

Calculates the average semantic synset overlap within a text, based on a list of lemma groups and a synset dictionary. The computation is normalized by the number of text segments.

```APIDOC
## overlap

### Description
Compute average overlap in a text. Computes semantic synset overlap (synonyms), based on a lemma list group and a dictionary containing synsets. Note that the computations are carried out dividing by number of text segments considered; matches TAACO implementation. For more details about this measurement, refer to [CKM16]

### Parameters
#### Path Parameters
- **lemma_list_group** (List of List of strings) - Required - List of tokenized and lemmatized sentences
- **synset_dict** (Python dict) - Required - key-value pairs for lemma-synonyms

### Returns
Average overlap between sentences

### Return Type
float
```

--------------------------------

### pronoun_density

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/givenness.html

Computes the pronoun density of a given SpaCy document. Pronoun density is a measure of text complexity, calculated as the ratio between third-person pronouns and the total number of words in the text.

```APIDOC
## pronoun_density

### Description
Compute pronoun density. This is a measurement of text complexity, in the sense that a text with a higher pronoun density will be more difficult to read than a text with lower pronoun density (due to inferences needed). The way this is computed is taking the ratio between third person pronouns and total words in the text.

### Parameters
#### Path Parameters
- **doc** (spacy.tokens.doc.Doc) - Required - Document to be processed.

### Returns
- **float** - Pronoun density
```

--------------------------------

### TRUNAJOD.utils.is_adverb

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Checks if a given token's part-of-speech tag is 'ADV'.

```APIDOC
## TRUNAJOD.utils.is_adverb

### Description
Return `True` if `pos_tag` is `ADV`, False otherwise.

### Parameters
#### Path Parameters
- **pos_tag** (string) – Part of Speech tag

### Returns
- True if POS is adverb (boolean)
```

--------------------------------

### TRUNAJOD.ttr.one_side_lexical_diversity_mtld

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/ttr.html

Calculates lexical diversity per MTLD in a single direction. It allows specifying the NLP model and the TTR segment threshold for computation.

```APIDOC
## TRUNAJOD.ttr.one_side_lexical_diversity_mtld

### Description
Lexical diversity per MTLD.

### Parameters
#### Parameters
- **doc** (_NLP Doc_) – Tokenized text
- **model_name** (_str_) – Determines which model is used (spacy or stanza), defaults to 'spacy'
- **ttr_segment** (_float_) – Threshold for TTR mean computation, defaults to 0.72

### Returns
- MLTD lexical diversity (float)
```

--------------------------------

### TRUNAJOD.surface_proxies.char_count

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Returns the total character count of a text, excluding punctuation and space tokens.

```APIDOC
## TRUNAJOD.surface_proxies.char_count

### Description
Return number of chars in a text. This count does not consider anything that its `Token.pos_` tag is either `PUNCT` or `SPACE`.

### Parameters
**doc** (_Spacy Doc_) – Text to be processed.

### Returns
Char count

### Return type
int
```

--------------------------------

### get_overall_markers

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/discourse_markers.html

Counts all types of discourse markers within a given Spacy Doc object.

```APIDOC
## TRUNAJOD.discourse_markers.get_overall_markers

### Description
Count all types of discourse markers.

### Parameters
#### Path Parameters
- **text** (spacy.tokens.doc.Doc) - Required - The text to be analized

### Returns
Average discourse markers over sentences

### Return type
float
```

--------------------------------

### TRUNAJOD.utils.is_noun

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Checks if a given token's part-of-speech tag is 'NOUN' or 'PROPN'.

```APIDOC
## TRUNAJOD.utils.is_noun

### Description
Return `True` if `pos_tag` is `NOUN` or `PROPN`, False otherwise.

### Parameters
#### Path Parameters
- **pos_tag** (string) – Part of Speech tag

### Returns
- True if POS is noun or proper noun (boolean)
```

--------------------------------

### TRUNAJOD.syllabizer.Syllabizer.number_of_syllables

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/syllabizer.html

Calculates and returns the number of syllables in a given Spanish word.

```APIDOC
## number_of_syllables

### Description
Return number of sillables of a word.

### Parameters
#### Path Parameters
- **word** (string) - Required - Word to be processed 

### Returns
Syllable count for the word. (int)
```

--------------------------------

### TRUNAJOD.surface_proxies.first_second_person_count

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Counts the occurrences of first and second person tokens within a processed text.

```APIDOC
## TRUNAJOD.surface_proxies.first_second_person_count

### Description
Count first|second person tokens.

### Parameters
**doc** (_Spacy Doc_) – Processed text

### Returns
First and second person count

### Return type
int
```

--------------------------------

### TRUNAJOD.utils.is_word

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/utils.html

Determines if a token is a word by checking if its part-of-speech tag is not punctuation, symbol, or space.

```APIDOC
## TRUNAJOD.utils.is_word

### Description
Return `True` if `pos_tag` is not punctuation, False otherwise. This method checks that the `pos_tag` does not belong to the following set: `{'PUNCT', 'SYM', 'SPACE'}`.

### Parameters
#### Path Parameters
- **pos_tag** (string) – Part of Speech tag

### Returns
- True if POS is a word (boolean)
```

--------------------------------

### get_local_coherence Function

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/entity_grid.html

Calculates the local coherence from an EntityGrid object using various heuristics described in [GS13].

```APIDOC
## Function `TRUNAJOD.entity_grid.get_local_coherence`

### Description
Get local coherence from entity grid. This method gets the coherence value using all the approaches described in [GS13], including: local_coherence_PU, local_coherence_PW, local_coherence_PACC, local_coherence_PU_dist, local_coherence_PW_dist, local_coherence_PACC_dist.

### Parameters
- **egrid** (EntityGrid) - Required - An EntityGrid object.

### Returns
- tuple of floats - Local coherence based on different heuristics.
```

--------------------------------

### word_count

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Returns the total number of words in a given text (Spacy Doc).

```APIDOC
## word_count

### Description
Return number of words in a text.

### Parameters
* **doc** (_Spacy Doc_) – Text to be processed.

### Returns
Word count

### Return type
int
```

--------------------------------

### sentence_count

Source: https://trunajod20.readthedocs.io/en/latest/api_reference/surface_proxies.html

Calculates and returns the number of sentences present in a given text (Spacy Doc).

```APIDOC
## sentence_count

### Description
Return number of sentences in a text.

### Parameters
* **doc** (_Spacy Doc_) – Text to be processed

### Returns
Number of sentences in the text

### Return type
int
```
Tokens:
Raw