### Installation and Setup

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/quickstart.md

Install the textacy library and download a spaCy language pipeline.

```APIDOC
## Installation and Setup

### Description
Install the textacy library and download a spaCy language pipeline for English.

### Method
Shell commands

### Endpoint
N/A

### Parameters
None

### Request Example
```shell
$ pip install textacy
$ python -m spacy download en_core_web_sm
```

### Response
None
```

--------------------------------

### Install Textacy and Language Models

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/quickstart.md

Commands to install the textacy library and download the necessary spaCy language pipeline for text processing.

```shell
pip install textacy
python -m spacy download en_core_web_sm
```

--------------------------------

### Text Extraction: Words and N-grams in Python

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Demonstrates how to extract words and n-grams from textacy-processed documents. It includes examples of default filtering (stop words, punctuation) and custom filtering based on Part-of-Speech tags, n-gram size, and minimum frequency.

```python
import textacy
from textacy import extract

doc = textacy.make_spacy_doc(
    "The quick brown fox jumps over the lazy dog near the old stone bridge.",
    lang="en_core_web_sm"
)

# Extract words with default filters (no stop words, no punctuation)
words = list(extract.words(doc))
print([w.text for w in words])
# ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog', 'near', 'old', 'stone', 'bridge']

# Extract words with custom POS filters
nouns = list(extract.words(doc, include_pos={"NOUN"}))
print([w.text for w in nouns])
# ['fox', 'dog', 'stone', 'bridge']

# Extract bigrams
bigrams = list(extract.ngrams(doc, n=2, filter_punct=True))
print([ng.text for ng in bigrams[:5]])
# ['quick brown', 'brown fox', 'fox jumps', 'lazy dog', 'old stone']

# Extract multiple n-gram sizes at once
ngrams = list(extract.ngrams(doc, n=(2, 3), filter_stops=True))
print([ng.text for ng in ngrams[:6]])
# ['quick brown', 'brown fox', 'fox jumps', 'lazy dog', 'old stone', 'stone bridge', ...]

# Extract n-grams by minimum frequency
bigrams_freq = list(extract.ngrams(doc, n=2, min_freq=2))
```

--------------------------------

### Compute String Similarity Metrics

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Provides examples for calculating edit-based, token-based, and hybrid string similarity. These metrics are useful for comparing text sequences, handling transpositions, and evaluating term overlap.

```python
from textacy import similarity

# Edit-based metrics
print(f"Levenshtein: {similarity.levenshtein('natural language processing', 'natural language understanding'):.4f}")
print(f"Jaro: {similarity.jaro('natural language processing', 'natural language understanding'):.4f}")

# Token-based metrics
seq1 = ["natural", "language", "processing", "is", "fascinating"]
seq2 = ["natural", "language", "understanding", "is", "exciting"]
print(f"Jaccard: {similarity.jaccard(seq1, seq2):.4f}")
print(f"Cosine: {similarity.cosine(seq1, seq2):.4f}")

# Hybrid metrics
print(f"Token sort ratio: {similarity.token_sort_ratio('New York City', 'City of New York'):.4f}")
print(f"Monge-Elkan: {similarity.monge_elkan(['machine', 'learning'], ['machine', 'method']):.4f}")
```

--------------------------------

### Compute Lexical Diversity Metrics in Python using textacy

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Provides Python code examples for calculating lexical diversity and vocabulary richness using the text_stats.diversity module in textacy. This helps in understanding the variety of words used in a text.

```Python
import textacy
from textacy import text_stats as ts

text = """
The dog chased the cat. The cat climbed the tree. The tree was tall.
The dog barked loudly. The cat meowed softly. They played all day.
"""
doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")


```

--------------------------------

### Initialize and Download CapitolWords Dataset

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/tutorials/tutorial-1.md

Initializes the CapitolWords dataset object and downloads the corpus to the local environment.

```python
import textacy.datasets
dataset = textacy.datasets.CapitolWords()
dataset.download()
```

--------------------------------

### Initialize and Fit NMF Topic Model

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/tutorials/tutorial-2.md

Demonstrates how to initialize a Non-negative Matrix Factorization (NMF) topic model using textacy.tm and fit it to a pre-computed document-term matrix.

```python
import textacy.tm
model = textacy.tm.TopicModel("nmf", n_topics=10)
model.fit(doc_term_matrix)
```

--------------------------------

### Text Extraction: Noun Chunks in Python

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Illustrates the extraction of noun chunks (noun phrases) from documents using textacy's `extract.noun_chunks` function. Examples show how to extract chunks with default settings and how to include determiners.

```python
import textacy
from textacy import extract

text = "The quick brown fox jumped over the lazy sleeping dog in the beautiful garden."
doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")

# Extract noun chunks
chunks = list(extract.noun_chunks(doc))
print([chunk.text for chunk in chunks])
# ['quick brown fox', 'lazy sleeping dog', 'beautiful garden']

# Keep determiners
chunks_with_det = list(extract.noun_chunks(doc, drop_determiners=False))
print([chunk.text for chunk in chunks_with_det])
# ['The quick brown fox', 'the lazy sleeping dog', 'the beautiful garden']
```

--------------------------------

### Initialize Corpus and Analyze Metadata

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/quickstart.md

Demonstrates creating a textacy Corpus from records and performing basic statistical analysis such as calculating sentence counts, median metadata values, and top lemma frequencies.

```python
import textacy
import statistics

records = [{"title": "The General in His Labyrinth", "pub_yr": 1989}]
corpus = textacy.Corpus("en_core_web_sm", records)

# Analyze corpus statistics
print(corpus.n_sents)
median_year = corpus.agg_metadata("pub_yr", statistics.median)
word_counts = corpus.word_counts(by="lemma_")
```

--------------------------------

### Text Preprocessing: Creating Pipelines in Python

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Shows how to create and use preprocessing pipelines in textacy by chaining multiple preprocessing functions. This allows for efficient, sequential application of various text transformations. Custom arguments can be passed using `functools.partial`.

```python
from functools import partial
from textacy import preprocessing

# Create a preprocessing pipeline
preproc = preprocessing.make_pipeline(
    preprocessing.replace.urls,
    preprocessing.replace.emails,
    preprocessing.replace.hashtags,
    preprocessing.replace.emojis,
    preprocessing.normalize.whitespace,
    preprocessing.remove.punctuation,
)

text = "Check out https://example.com! 🎉 Contact: test@email.com #awesome"
print(preproc(text))
# "_URL_  _EMOJI_ Contact  _EMAIL_ _TAG_"

# Pipeline with custom arguments using functools.partial
preproc_custom = preprocessing.make_pipeline(
    partial(preprocessing.remove.punctuation, only=[ ".", "!", "?" ]),
    partial(preprocessing.replace.numbers, repl="<NUM>"),
    preprocessing.normalize.whitespace,
)

text = "Price: $42.99! Quantity: 100?"
print(preproc_custom(text))
# "Price: $<NUM> Quantity: <NUM>"
```

--------------------------------

### Computing Text Statistics with textacy

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Illustrates how to compute various statistical measures for a document using the textacy library's text_stats module. This includes basic counts like the number of words, readability scores such as the Flesch-Kincaid grade level, and lexical diversity metrics like the Type-Token Ratio (TTR). It also shows how to get part-of-speech tag counts.

```python
from textacy import text_stats as ts

# Basic statistics
ts.basics.n_words(doc)

# Readability statistics
ts.readability.flesch_kincaid_grade_level(doc)

# Lexical diversity statistics
ts.diversity.ttr(doc)

# Part-of-speech counts
ts.counts.pos(doc)
```

--------------------------------

### Create spaCy Doc with Custom spaCy Pipeline

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Demonstrates creating a spaCy 'Doc' object by first loading a spaCy language pipeline and then passing it to textacy's `make_spacy_doc`. This allows for customization, such as disabling specific pipeline components like the parser.

```python
en = textacy.load_spacy_lang("en_core_web_sm", disable=("parser",))
doc = textacy.make_spacy_doc(text, lang=en)
doc._.preview
```

--------------------------------

### Text File I/O with Textacy

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Demonstrates reading and writing text, JSON/JSONL, and CSV files using textacy.io, including support for compressed files.

```python
import textacy.io as tio

# Read lines from a text file
texts = list(tio.read_text("documents.txt", lines=True))

# Read JSON/JSONL files
records = list(tio.read_json("data.jsonl", lines=True))

# Read CSV files
rows = list(tio.read_csv("data.csv"))

# Write text to file
tio.write_text(["Line 1", "Line 2", "Line 3"], "output.txt", lines=True)

# Write JSON/JSONL
tio.write_json([{"text": "doc1"}, {"text": "doc2"}], "output.jsonl", lines=True)

# Work with compressed files (automatically detected by extension)
texts_gz = list(tio.read_text("documents.txt.gz", lines=True))
tio.write_text(texts, "output.txt.gz", lines=True)
```

--------------------------------

### Create and Analyze spaCy Documents

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/quickstart.md

Demonstrates how to convert raw text into a spaCy Doc object and perform basic linguistic analysis such as entity extraction, SVO triple identification, and text statistics.

```python
import textacy
from textacy import extract, text_stats as ts

text = "Many years later..."
doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")

# Extract entities and SVO triples
entities = list(extract.entities(doc, include_types={"PERSON", "LOCATION"}))
svos = list(extract.subject_verb_object_triples(doc))

# Calculate statistics
n_words = ts.n_words(doc)
diversity = ts.diversity.ttr(doc)
grade_level = ts.flesch_kincaid_grade_level(doc)
```

--------------------------------

### Preprocessing Text with Textacy Pipelines

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/tutorials/tutorial-1.md

Demonstrates how to create a preprocessing pipeline using textacy.preproc to normalize unicode, quotation marks, and whitespace in text records.

```python
preprocessor = preproc.make_pipeline(
    preproc.normalize.unicode,
    preproc.normalize.quotation_marks,
    preproc.normalize.whitespace,
)
preproc_text = preprocessor(record.text)
```

--------------------------------

### Using textacy Datasets

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Shows how to use pre-implemented textacy Dataset classes to download and access specific datasets, like Capitol Words, and retrieve records.

```APIDOC
## Using textacy Datasets

### Description
This section explains how to leverage built-in `Dataset` classes in textacy to easily download and access common datasets. It demonstrates initializing a dataset, downloading its data, and retrieving records with specific filters.

### Method
`textacy.datasets.<DatasetName>()` and its methods (`download()`, `records()`, `texts()`)

### Endpoint
N/A (Dataset Management)

### Parameters
#### Initialization Parameters (Example: CapitolWords)
- **path** (string) - Optional - Directory to store dataset files. Defaults to a textacy data directory.

#### `records()` Parameters (Example: CapitolWords)
- **speaker_name** (set of strings) - Optional - Filter records by speaker name.
- **speaker_party** (set of strings) - Optional - Filter records by speaker party.
- **chamber** (set of strings) - Optional - Filter records by legislative chamber.
- **date_range** (tuple of strings) - Optional - Filter records by date range (YYYY-MM-DD).
- **limit** (int) - Optional - Maximum number of records to return.

### Request Example
```python
import textacy.datasets

# Initialize the CapitolWords dataset
ds = textacy.datasets.CapitolWords()

# Download the dataset if not already present
ds.download()

# Retrieve records filtered by speaker name
records = ds.records(speaker_name={"Hillary Clinton", "Barack Obama"})

# Get the next record from the iterator
first_record = next(records)
print(first_record)
```

### Response
#### Success Response (Tuple)
A tuple containing the text and its associated metadata dictionary.

#### Response Example
```
('I yield myself 15 minutes of the time controlled by the Democrats.',
 {'date': '2001-02-13',
  'congress': 107,
  'speaker_name': 'Hillary Clinton',
  'speaker_party': 'D',
  'title': 'MORNING BUSINESS',
  'chamber': 'Senate'})
```
```

--------------------------------

### Stream text from disk

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Demonstrates reading raw text files line-by-line and converting them into spaCy documents using textacy.io.read_text.

```python
texts = textacy.io.read_text('~/Desktop/burton-tweets.txt', lines=True)
for text in texts:
    doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")
    print(doc._.preview)
```

--------------------------------

### Compare Documents using Similarity Metrics

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/quickstart.md

Shows how to compare two documents using Levenshtein distance, cosine similarity on lemmas, and readability comparisons.

```python
from textacy import similarity, extract

other_doc = textacy.make_spacy_doc("Finally, one Tuesday...", lang="en_core_web_sm")

# Levenshtein distance
lev_dist = similarity.levenshtein(doc.text, other_doc.text)

# Cosine similarity on lemmas
cos_sim = similarity.cosine(
    (tok.lemma_ for tok in extract.words(doc)),
    (tok.lemma_ for tok in extract.words(other_doc))
)
```

--------------------------------

### Topic Modeling with Textacy

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Demonstrates training a topic model (NMF) on a corpus, vectorizing documents, and extracting top terms and documents per topic. Includes saving and loading the model.

```python
import textacy
from textacy import extract
from textacy.topic_modeling import TopicModel
from textacy.vector import Vectorizer

# Prepare corpus
texts = [
    "Machine learning algorithms process large datasets.",
    "Neural networks learn patterns from data.",
    "Climate change affects global weather patterns.",
    "Renewable energy reduces carbon emissions.",
    "Deep learning models require GPU computing.",
    "Solar and wind power are sustainable energy sources.",
]
corpus = textacy.Corpus("en_core_web_sm", data=texts)

# Vectorize
tokenized_docs = [
    [term.lemma_ for term in extract.terms(doc, ngs=1)]
    for doc in corpus
]
vectorizer = Vectorizer(tf_type="linear", idf_type="smooth", norm="l2", min_df=1)
doc_term_matrix = vectorizer.fit_transform(tokenized_docs)

# Train topic model
model = TopicModel("nmf", n_topics=2)  # Options: "nmf", "lda", "lsa"
model.fit(doc_term_matrix)

# Transform documents to topic space
doc_topic_matrix = model.transform(doc_term_matrix)
print(f"Doc-topic matrix shape: {doc_topic_matrix.shape}")

# Get top terms for each topic
for topic_idx, top_terms in model.top_topic_terms(vectorizer.id_to_term, top_n=5):
    print(f"Topic {topic_idx}: {', '.join(top_terms)}")

# Get top documents for each topic
for topic_idx, doc_indices in model.top_topic_docs(doc_topic_matrix, top_n=2):
    print(f"Topic {topic_idx} top docs: {list(doc_indices)}")

# Get topic weights (importance across corpus)
for i, weight in enumerate(model.topic_weights(doc_topic_matrix)):
    print(f"Topic {i} weight: {weight:.4f}")

# Save and load model
model.save("topic_model.pkl")
# loaded_model = TopicModel.load("topic_model.pkl")
```

--------------------------------

### Topic Modeling with Textacy

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Demonstrates how to prepare a corpus, vectorize it, train a topic model (NMF, LDA, or LSA), and extract topic-related information such as top terms and documents.

```APIDOC
## Topic Modeling with Textacy

### Description
This example shows how to prepare a corpus, vectorize it using a specified TF-IDF scheme, train a topic model (NMF, LDA, or LSA), and then extract various topic-related insights.

### Method
Not applicable (Python script)

### Endpoint
Not applicable (Python script)

### Parameters
None

### Request Example
```python
import textacy
from textacy import extract
from textacy.topic_modeling import TopicModel
from textacy.vector import Vectorizer

# Prepare corpus
texts = [
    "Machine learning algorithms process large datasets.",
    "Neural networks learn patterns from data.",
    "Climate change affects global weather patterns.",
    "Renewable energy reduces carbon emissions.",
    "Deep learning models require GPU computing.",
    "Solar and wind power are sustainable energy sources.",
]
coprus = textacy.Corpus("en_core_web_sm", data=texts)

# Vectorize
tokenized_docs = [
    [term.lemma_ for term in extract.terms(doc, ngs=1)]
    for doc in corpus
]
vectorizer = Vectorizer(tf_type="linear", idf_type="smooth", norm="l2", min_df=1)
doc_term_matrix = vectorizer.fit_transform(tokenized_docs)

# Train topic model
model = TopicModel("nmf", n_topics=2)  # Options: "nmf", "lda", "lsa"
model.fit(doc_term_matrix)

# Transform documents to topic space
doc_topic_matrix = model.transform(doc_term_matrix)
print(f"Doc-topic matrix shape: {doc_topic_matrix.shape}")

# Get top terms for each topic
for topic_idx, top_terms in model.top_topic_terms(vectorizer.id_to_term, top_n=5):
    print(f"Topic {topic_idx}: {', '.join(top_terms)}")

# Get top documents for each topic
for topic_idx, doc_indices in model.top_topic_docs(doc_topic_matrix, top_n=2):
    print(f"Topic {topic_idx} top docs: {list(doc_indices)}")

# Get topic weights (importance across corpus)
for i, weight in enumerate(model.topic_weights(doc_topic_matrix)):
    print(f"Topic {i} weight: {weight:.4f}")

# Save and load model
model.save("topic_model.pkl")
# loaded_model = TopicModel.load("topic_model.pkl")
```

### Response
#### Success Response (200)
Prints the shape of the doc-topic matrix, top terms for each topic, top documents for each topic, and topic weights.

#### Response Example
```
Doc-topic matrix shape: (6, 2)
Topic 0: learning, models, deep, machine, data
Topic 1: energy, power, climate, change, affects
Topic 0 top docs: [0, 4]
Topic 1 top docs: [2, 3]
Topic 0 weight: 0.5000
Topic 1 weight: 0.5000
```
```

--------------------------------

### Inspect Vectorizer Weighting Schemes

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/changes.md

Demonstrates how to inspect the mathematical weighting formula used by the Vectorizer class. This is useful for understanding how document-term matrix values are calculated based on different configuration parameters.

```python
from textacy.vsm import Vectorizer

# Simple weighting case
print(Vectorizer(apply_idf=True, idf_type='smooth').weighting)

# Complex BM25 weighting case
print(Vectorizer(tf_type='bm25', apply_idf=True, idf_type='smooth', apply_dl=True).weighting)
```

--------------------------------

### Creating spaCy Documents from Textacy

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/tutorials/tutorial-1.md

Shows how to convert preprocessed text and metadata into a spaCy Doc object using a specific language model.

```python
doc = textacy.make_spacy_doc((preproc_text, record.meta), lang="en_core_web_sm")
```

--------------------------------

### POST /preprocessing/pipeline

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/api_reference/preprocessing.rst

Creates a processing pipeline by chaining multiple text transformation functions together.

```APIDOC
## POST /preprocessing/pipeline

### Description
Creates a callable pipeline that applies a sequence of preprocessing functions to a given text input.

### Method
POST

### Endpoint
/preprocessing/pipeline

### Parameters
#### Request Body
- **functions** (list) - Required - A list of preprocessing functions to be applied in order.

### Request Example
{
  "functions": ["normalize.whitespace", "remove.punctuation"]
}

### Response
#### Success Response (200)
- **pipeline** (callable) - The resulting pipeline function.

#### Response Example
{
  "status": "success",
  "message": "Pipeline created successfully"
}
```

--------------------------------

### Preprocess Text Data

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/tutorials/tutorial-1.md

Demonstrates how to use textacy.preprocessing to normalize text, such as replacing numeric values with a placeholder constant.

```python
from textacy import preprocessing as preproc
cleaned_text = preproc.replace.numbers(record.text)
```

--------------------------------

### Create Text Preprocessing Pipeline

Source: https://github.com/chartbeat-labs/textacy/blob/main/CHANGES.md

Demonstrates the use of the new make_pipeline function to chain multiple text preprocessing operations sequentially into a single callable object.

```python
from textacy import preprocessing

# Create a pipeline to normalize whitespace and remove HTML tags
preprocess = preprocessing.make_pipeline(
    preprocessing.normalize.whitespace,
    preprocessing.remove.html_tags
)

cleaned_text = preprocess("  <p>Hello   World!</p>  ")
```

--------------------------------

### Text Preprocessing: Basic Replacements in Python

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Demonstrates basic text preprocessing functions in textacy for replacing common patterns like numbers, emojis, hashtags, user handles, and URLs. These functions are useful for normalizing text data before further analysis.

```python
from textacy import preprocessing

text = "The price is $42.99 and quantity is 100."
print(preprocessing.replace.numbers(text))
# "The price is $_NUMBER_ and quantity is _NUMBER_."

text = "Great job! 🎉👍"
print(preprocessing.replace.emojis(text))
# "Great job! _EMOJI__EMOJI_"

text = "Loving this #Python #NLP tutorial"
print(preprocessing.replace.hashtags(text))
# "Loving this _TAG_ _TAG_ tutorial"

text = "Thanks @textacy_dev for the great library!"
print(preprocessing.replace.user_handles(text))
# "Thanks _USER_ for the great library!"

print(preprocessing.replace.urls(text, repl="<LINK>"))
```

--------------------------------

### Preprocess and Normalize Text using textacy

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Shows how to clean and standardize text data using textacy's preprocessing functions. This includes removing punctuation and normalizing whitespace, which are common steps before text analysis. It requires importing the 'preprocessing' sub-package.

```python
from textacy import preprocessing
preprocessing.normalize.whitespace(preprocessing.remove.punctuation(text))[:80]
```

--------------------------------

### Manage Document Collections with Corpus

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Shows how to initialize a textacy Corpus object from raw text or metadata-rich records. It also covers accessing corpus statistics and performing basic document iteration and indexing.

```python
import textacy

# Initialize corpus from text list
texts = ["NLP is a subfield of AI.", "Machine learning powers NLP."]
corpus = textacy.Corpus("en_core_web_sm", data=texts)

# Access statistics
print(f"Documents: {corpus.n_docs}")
print(f"Tokens: {corpus.n_tokens}")

# Add documents to existing corpus
corpus.add("This is another document about AI.")

# Iterate over documents
for doc in corpus:
    print(doc._.preview)
```

--------------------------------

### Similarity Metrics Overview

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/api_reference/similarity.rst

Overview of the available similarity functions provided by the textacy.similarity module.

```APIDOC
## Similarity Metrics API

### Description
The textacy.similarity module provides functions to compute similarity scores between strings, tokens, or sequences. These are categorized into four main groups: edit-based, token-based, sequence-based, and hybrid.

### Available Methods
- **Edit-based**: hamming, levenshtein, jaro, character_ngrams
- **Token-based**: jaccard, sorensen_dice, tversky, cosine, bag
- **Sequence-based**: matching_subsequences_ratio
- **Hybrid**: token_sort_ratio, monge_elkan

### Usage Example
```python
from textacy import similarity
score = similarity.tokens.jaccard(tokens1, tokens2)
```

### Parameters
- **input1** (str/list) - Required - The first sequence or set of tokens.
- **input2** (str/list) - Required - The second sequence or set of tokens.

### Response
- **score** (float) - A similarity score typically ranging from 0.0 to 1.0.
```

--------------------------------

### Creating a textacy Corpus

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Explains how to create a textacy Corpus, which is an ordered collection of spaCy Docs processed by the same language pipeline. It shows different ways to populate a corpus.

```APIDOC
## Creating a textacy Corpus

### Description
A `textacy.Corpus` is an ordered collection of spaCy `Doc` objects, all processed using the same language pipeline. This section details how to instantiate a Corpus and populate it with data from various sources, such as iterators of records or texts.

### Method
`textacy.Corpus()`

### Endpoint
N/A (Object Instantiation)

### Parameters
#### Initialization Parameters
- **lang** (string or spaCy language object) - Required - The language and pipeline to use for processing documents. Can be a language code (e.g., "en") or a loaded spaCy language object.
- **data** (iterable) - Optional - An iterable of texts, records, or spaCy `Doc` objects to populate the corpus. If not provided, an empty corpus is created.
- **docs** (iterable) - Optional - An iterable of pre-processed spaCy `Doc` objects.
- **meta_merge_strategy** (string) - Optional - Strategy for merging metadata when adding documents. Defaults to 'union'.

### Request Example
```python
import textacy
import textacy.datasets

# Example 1: Creating a corpus from a dataset iterator
# Ensure the dataset is downloaded first
ds = textacy.datasets.CapitolWords()
ds.download()
records = ds.records(speaker_party="R", chamber="House", limit=100)

# Create a corpus using a specified language pipeline
capsule_corpus = textacy.Corpus("en", data=records)
print(capsule_corpus)

# Example 2: Creating a corpus with a pre-loaded spaCy pipeline and specific texts
snlp = textacy.load_spacy_lang("en_core_web_sm", disable=("parser", "tagger"))
text_data = ["This is the first document.", "This is the second document."]
text_corpus = textacy.Corpus(snlp, data=text_data)
print(text_corpus)
```

### Response
#### Success Response (Corpus Object)
A `textacy.Corpus` object containing the processed documents.

#### Response Example
```
Corpus(100 docs, 31356 tokens)
Corpus(2 docs, 12 tokens)
```

### Usage Examples
```python
# Accessing documents by index
print(corpus[-1]._.preview)

# Slicing the corpus
for doc in corpus[10:15]:
    print(doc._.preview)

# Filtering documents using a lambda function
obama_docs = list(corpus.get(lambda doc: doc._.meta["speaker_name"] == "Barack Obama"))
print(f"Found {len(obama_docs)} documents by Barack Obama.")
```

**Note:** All data in a `textacy.Corpus` is stored in-memory. Ensure sufficient RAM for large datasets.
```

--------------------------------

### Extract Keyterms using YAKE Algorithm in Python

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Demonstrates how to extract key terms from text using the YAKE algorithm in Python with textacy. It shows default usage and customization of parameters like normalization, n-grams, part-of-speech tags, and top-n selection. Lower scores indicate more important terms.

```Python
import textacy
from textacy import extract

text = """
Artificial intelligence is rapidly advancing. Machine learning, deep learning,
and neural networks are key technologies. Natural language processing and
computer vision are important AI application areas. Data science combines
statistics and machine learning for insights.
"""
doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")

# Extract keyterms using YAKE (lower scores = more important)
keyterms = extract.keyterms.yake(doc, normalize="lemma", topn=10)
print("YAKE keyterms:")
for term, score in keyterms:
    print(f"  {term}: {score:.4f}")

# Customize YAKE parameters
keyterms_custom = extract.keyterms.yake(
    doc,
    normalize="lower",
    ngrams=(1, 2, 3),  # Consider unigrams, bigrams, trigrams
    include_pos=("NOUN", "PROPN", "ADJ"),
    window_size=2,
    topn=0.1  # Top 10% of candidates
)
```

--------------------------------

### I/O: Reading and Writing Text Files

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Utilities for reading and writing text data in various formats, including plain text, JSON, and CSV, with support for compressed files.

```APIDOC
## I/O: Reading and Writing Text Files

### Description
The `io` module provides utilities for reading and writing text data in various formats.

### Method
Not applicable (Python script)

### Endpoint
Not applicable (Python script)

### Parameters
None

### Request Example
```python
import textacy.io as tio

# Read lines from a text file
texts = list(tio.read_text("documents.txt", lines=True))

# Read JSON/JSONL files
records = list(tio.read_json("data.jsonl", lines=True))

# Read CSV files
rows = list(tio.read_csv("data.csv"))

# Write text to file
tio.write_text(["Line 1", "Line 2", "Line 3"], "output.txt", lines=True)

# Write JSON/JSONL
tio.write_json([{"text": "doc1"}, {"text": "doc2"}], "output.jsonl", lines=True)

# Work with compressed files (automatically detected by extension)
texts_gz = list(tio.read_text("documents.txt.gz", lines=True))
tio.write_text(texts, "output.txt.gz", lines=True)
```

### Response
#### Success Response (200)
Successfully reads from and writes to specified files in various formats (text, JSON, CSV), including compressed files.

#### Response Example
No direct output to display, as the function performs file operations.
```

--------------------------------

### Create and query a textacy Corpus

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Covers the instantiation of a Corpus object from data streams and demonstrates how to perform indexing and boolean filtering on the collection.

```python
corpus = textacy.Corpus("en", data=records)
# Filtering with lambda
obama_docs = list(corpus.get(lambda doc: doc._.meta["speaker_name"] == "Barack Obama"))
# Creating with specific spaCy pipeline
corpus = textacy.Corpus(textacy.load_spacy_lang("en_core_web_sm", disable=("parser", "tagger")), data=ds.texts(speaker_party="R", chamber="House", limit=100))
```

--------------------------------

### Create spaCy Doc from Text with textacy

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Illustrates the creation of a spaCy 'Doc' object from raw text using textacy's `make_spacy_doc` function. This process involves loading a specified spaCy language model (e.g., 'en_core_web_sm') for tokenization and linguistic annotation.

```python
doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")
doc._.preview
```

--------------------------------

### Read and Write CSV with Quoting Options

Source: https://github.com/chartbeat-labs/textacy/blob/main/CHANGES.md

Demonstrates how to handle problematic CSV files by specifying quoting options during read and write operations.

```python
import csv
from textacy.io import read_csv, write_csv

# Reading a CSV with specific quoting
data = read_csv("input.csv", quoting=csv.QUOTE_ALL)

# Writing a CSV with specific quoting
write_csv(data, "output.csv", quoting=csv.QUOTE_MINIMAL)
```

--------------------------------

### Stream JSON records from disk

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Shows how to read compressed JSON lines files containing both text and metadata, and process them into spaCy documents.

```python
records = textacy.io.read_json("textacy/data/capitol_words/capitol-words-py3.json.gz", mode="rt", lines=True)
for record in records:
    doc = textacy.make_spacy_doc((record["text"], {"title": record["title"]}), lang="en_core_web_sm")
    print(doc._.preview)
    print("meta:", doc._.meta)
    break
```

--------------------------------

### POST /preprocessing/normalize

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/api_reference/preprocessing.rst

Normalization utilities to standardize text formats such as whitespace, unicode, and quotation marks.

```APIDOC
## POST /preprocessing/normalize

### Description
Standardizes various text elements to ensure consistency across datasets.

### Method
POST

### Endpoint
/preprocessing/normalize/{type}

### Parameters
#### Path Parameters
- **type** (string) - Required - The normalization type (e.g., 'whitespace', 'unicode', 'bullet_points').

#### Request Body
- **text** (string) - Required - The raw text to normalize.

### Request Example
{
  "text": "Hello   World!"
}

### Response
#### Success Response (200)
- **normalized_text** (string) - The cleaned text string.

#### Response Example
{
  "normalized_text": "Hello World!"
}
```

--------------------------------

### Identifying Key Terms with textacy

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Shows how to identify key terms within a document using TextRank and SG-Rank algorithms provided by textacy. These methods help in discovering the most important concepts or phrases in a text, with options for normalization and specifying the number of terms.

```python
from textacy import extract

# Using TextRank to find key terms
extract.keyterms.textrank(doc, normalize="lemma", topn=10)

# Using SG-Rank to find key terms
extract.keyterms.sgrank(doc, ngrams=(1, 2, 3, 4), normalize="lower", topn=0.1)
```

--------------------------------

### Corpus: Saving and Loading

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Explains how to save corpus data to disk for persistence and reload it later.

```APIDOC
## Corpus: Saving and Loading

### Description
Save corpus data to disk in a binary format for efficient storage and later retrieval.

### Method
N/A (Illustrative Python code)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
import textacy

# Create and populate corpus
texts = ["Document one.", "Document two.", "Document three."]
coprus = textacy.Corpus("en_core_web_sm", data=texts)

# Save to binary format (efficient, preserves annotations)
coprus.save("./my_corpus.bin.gz")

# Load from disk
loaded_corpus = textacy.Corpus.load("en_core_web_sm", "./my_corpus.bin.gz")
print(loaded_corpus)
```

### Response
N/A (Illustrative Python output)

#### Success Response (200)
N/A

#### Response Example
```
Corpus(docs=[Doc(text='Document one.', ...), Doc(text='Document two.', ...), Doc(text='Document three.', ...)])
```
```

--------------------------------

### Define Sample Text

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Defines a multi-line string variable containing sample text to be used for demonstrations. This text is representative of natural language data.

```python
text = (
    "Since the so-called \"statistical revolution\" in the late 1980s and mid 1990s, "
    "much Natural Language Processing research has relied heavily on machine learning. "
    "Formerly, many language-processing tasks typically involved the direct hand coding "
    "of rules, which is not in general robust to natural language variation. "
    "The machine-learning paradigm calls instead for using statistical inference "
    "to automatically learn such rules through the analysis of large corpora "
    "of typical real-world examples."
)
```

--------------------------------

### Extract Keyterms using SGRank Algorithm in Python

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Illustrates how to extract key terms from text using the SGRank algorithm with textacy in Python. It covers basic extraction and customization of parameters such as n-grams, normalization, and top-n selection. SGRank utilizes statistical and graph-based features.

```Python
import textacy
from textacy import extract

text = """
Climate change poses significant challenges to global ecosystems. Rising
temperatures affect biodiversity and weather patterns. Renewable energy
sources like solar and wind power can help reduce carbon emissions.
International cooperation is essential for addressing climate change.
"""
doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")

# Extract keyterms using SGRank
keyterms = extract.keyterms.sgrank(doc, normalize="lemma", topn=10)
print("SGRank keyterms:")
for term, score in keyterms:
    print(f"  {term}: {score:.4f}")

# Customize SGRank parameters
keyterms_custom = extract.keyterms.sgrank(
    doc,
    ngrams=(1, 2, 3, 4),
    normalize="lower",
    topn=0.1
)
```

--------------------------------

### Extract Keywords in Context using textacy

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Demonstrates how to extract keywords from a given text and display them within their surrounding context. This function helps in understanding word usage within a document. It requires importing the 'extract' module from textacy.

```python
from textacy import extract
list(extract.keyword_in_context(text, "language", window_width=25, pad_context=True))
```

--------------------------------

### Calculate Readability Metrics in Python with textacy

Source: https://context7.com/chartbeat-labs/textacy/llms.txt

Demonstrates the computation of various readability scores using textacy's text_stats.readability module in Python. It includes metrics like Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, ARI, SMOG, and LIX.

```Python
import textacy
from textacy import text_stats as ts

text = """
The implementation of sophisticated natural language processing algorithms
requires extensive computational resources. Advanced neural architectures
demonstrate unprecedented performance on complex linguistic tasks.
"""
doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")

# Flesch Reading Ease (higher = easier, 0-100 scale)
print(f"Flesch Reading Ease: {ts.flesch_reading_ease(doc):.2f}")

# Flesch-Kincaid Grade Level (US grade level)
print(f"Flesch-Kincaid Grade: {ts.flesch_kincaid_grade_level(doc):.2f}")

# Gunning Fog Index (years of education needed)
print(f"Gunning Fog Index: {ts.gunning_fog_index(doc):.2f}")

# Coleman-Liau Index
print(f"Coleman-Liau Index: {ts.coleman_liau_index(doc):.2f}")

# Automated Readability Index
print(f"Automated Readability Index: {ts.automated_readability_index(doc):.2f}")

# SMOG Index (for medical/technical texts)
print(f"SMOG Index: {ts.smog_index(doc):.2f}")

# LIX (Swedish readability measure)
print(f"LIX: {ts.lix(doc):.2f}")

# Language-specific measures
# For German texts:
# ts.wiener_sachtextformel(doc, variant=1)

# For Italian texts:
# ts.gulpease_index(doc)

# For Spanish texts:
# ts.perspicuity_index(doc)
# ts.mu_legibility_index(doc)
```

--------------------------------

### Managing and Analyzing a Textacy Corpus

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/tutorials/tutorial-1.md

Illustrates how to load multiple records into a Corpus object, aggregate metadata, and perform lemmatized pattern matching across the entire collection.

```python
corpus = textacy.Corpus("en_core_web_sm", data=preproc_records)
corpus.agg_metadata("speaker_name", collections.Counter)
matches = itertools.chain.from_iterable(extract.token_matches(doc, patterns) for doc in corpus)
collections.Counter(match.lemma_ for match in matches).most_common(20)
```

--------------------------------

### Reading Text from Files

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Demonstrates how to read text documents from a file, where each line represents a separate text document. It then processes these texts into spaCy Docs.

```APIDOC
## Reading Text from Files

### Description
This section shows how to read text data from a file where each line is a distinct text document. It then iterates through these texts, converting each into a spaCy Doc for further analysis.

### Method
`textacy.io.read_text()`

### Endpoint
N/A (File I/O)

### Parameters
#### Query Parameters
- **path** (string) - Required - The path to the text file.
- **lines** (boolean) - Optional - If True, reads the file line by line, treating each line as a separate text document. Defaults to False.

### Request Example
```python
import textacy

# Assuming '~/Desktop/burton-tweets.txt' contains one tweet per line
texts = textacy.io.read_text('~/Desktop/burton-tweets.txt', lines=True)

for text in texts:
    doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")
    print(doc._.preview)
```

### Response
#### Success Response (Iterator)
An iterator yielding strings, where each string is a line from the input file.

#### Response Example
```
Doc(32 tokens; "I love Daylight Savings Time: It's a biannual o...")
Doc(28 tokens; "Somewhere between \"this is irritating but meh\" ...")
Doc(20 tokens; "Spent an entire day translating structured data...")
```
```

--------------------------------

### Creating a spaCy Document

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/quickstart.md

Create a spaCy `Doc` object from a given text using a specified language model.

```APIDOC
## Creating a spaCy Document

### Description
Create a spaCy `Doc` object from a given text using a specified language model.

### Method
Python function call

### Endpoint
N/A

### Parameters
- **text** (string) - Required - The input text to process.
- **lang** (string) - Required - The language model to use (e.g., "en_core_web_sm").

### Request Example
```python
import textacy

text = (
    "Many years later, as he faced the firing squad, Colonel Aureliano Buendía "
    "was to remember that distant afternoon when his father took him to discover ice. "
    "At that time Macondo was a village of twenty adobe houses, built on the bank "
    "of a river of clear water that ran along a bed of polished stones, which were "
    "white and enormous, like prehistoric eggs. The world was so recent "
    "that many things lacked names, and in order to indicate them it was necessary to point."
)
doc = textacy.make_spacy_doc(text, lang="en_core_web_sm")
print(doc._.preview)
```

### Response
#### Success Response (200)
- **doc._.preview** (string) - A preview of the created spaCy Doc object.
```

--------------------------------

### Load Blank spaCy Language Pipeline

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/changes.md

Enables loading of blank spaCy Language pipelines for tokenization-only use cases. This is useful when annotations from a full spaCy model are not required, and it is disabled by default to prevent unexpected behavior.

```python
from textacy.load import load_spacy_lang

# Load a blank English language pipeline
nlp = load_spacy_lang("en_core_web_sm", allow_blank=True)
```

--------------------------------

### Create spaCy Doc with Metadata using textacy

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Shows how to create a spaCy 'Doc' object from text that includes associated metadata (like title, URL, source). The metadata is passed as a tuple along with the text and is accessible via the '._.meta' attribute of the Doc object.

```python
metadata = {
    "title": "Natural-language processing",
    "url": "https://en.wikipedia.org/wiki/Natural-language_processing",
    "source": "wikipedia",
}
doc = textacy.make_spacy_doc((text, metadata), lang="en_core_web_sm")
doc._.meta["title"]
```

--------------------------------

### Use built-in Dataset classes

Source: https://github.com/chartbeat-labs/textacy/blob/main/docs/source/walkthrough.md

Utilizes textacy's pre-implemented dataset classes to download and filter records efficiently.

```python
import textacy.datasets
ds = textacy.datasets.CapitolWords()
ds.download()
records = ds.records(speaker_name={"Hillary Clinton", "Barack Obama"})
next(records)
```