### Install Dependencies with Conda

Source: https://github.com/derwenai/pytextrank/blob/main/docs/setup.md

Create and activate a conda environment, then install project dependencies. This method uses an environment file for setup.

```bash
conda env create -f environment.yml
conda activate pytextrank
python3 -m spacy download en_core_web_sm
```

--------------------------------

### Install Dependencies with Pip

Source: https://github.com/derwenai/pytextrank/blob/main/docs/setup.md

Create a virtual environment and install project dependencies using pip. Ensure you activate the virtual environment before installing.

```bash
python3 -m venv venv
source venv/bin/activate
python3 -m pip install -U pip
python3 -m pip install -r requirements.txt
python3 -m spacy download en_core_web_sm
```

--------------------------------

### Install Altair

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Installs the Altair library, which is used for creating interactive visualizations.

```python
!pip install "altair"
```

--------------------------------

### Install JupyterLab with Pip

Source: https://github.com/derwenai/pytextrank/blob/main/docs/setup.md

Install JupyterLab using pip. If using user-level installation, ensure the user bin directory is added to your PATH.

```bash
python3 -m pip install jupyterlab
```

--------------------------------

### Install GraphViz library

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Install the `graphviz` Python package using pip. This package is necessary for rendering DOT files.

```bash
!pip install graphviz
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Installs the necessary development dependencies for the PyTextRank library. Run this command in your local environment to set up for development.

```bash
python3 -m pip install -r requirements-dev.txt
```

--------------------------------

### Preview Documentation Locally

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Starts a local web server to preview the generated documentation. Browse to http://localhost:8000 to view the site.

```bash
./bin/preview.py
```

--------------------------------

### Load Example Text

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Reads text content from a specified file path. Ensure the file exists at the given relative path.

```python
text = pathlib.Path("../dat/cfc.txt").read_text()
text
```

--------------------------------

### Install PyTextRank Dependencies with Conda

Source: https://github.com/derwenai/pytextrank/blob/main/README.md

Alternatively, set up the environment and install dependencies using Conda with the provided environment.yml file.

```bash
conda env create -f environment.yml
conda activate pytextrank
```

--------------------------------

### Install PyTextRank and spaCy Model

Source: https://github.com/derwenai/pytextrank/blob/main/README.md

Install the pytextrank package and a spaCy English model using pip. This is the first step to using PyTextRank.

```bash
python3 -m pip install pytextrank
python3 -m spacy download en_core_web_sm
```

--------------------------------

### Install PyTextRank Dependencies from Requirements

Source: https://github.com/derwenai/pytextrank/blob/main/README.md

If working directly from the Git repository, install dependencies using the requirements.txt file.

```bash
python3 -m pip install -r requirements.txt
```

--------------------------------

### Install JupyterLab with Conda

Source: https://github.com/derwenai/pytextrank/blob/main/docs/setup.md

Install JupyterLab using conda from the conda-forge channel.

```bash
conda install -c conda-forge jupyterlab
```

--------------------------------

### Example Output of Phrase Analysis

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

This section shows example output from the PyTextRank library, detailing the rank, count, text, and chunks for various extracted phrases.

```text
ic| phrase.rank: 0.18359439311764025
    phrase.count: 1
    phrase.text: 'mixed types'
ic| phrase.chunks: [mixed types]
ic| phrase.rank: 0.1784796193107821
    phrase.count: 3
    phrase.text: 'systems'
ic| phrase.chunks: [systems, systems, systems]
ic| phrase.rank: 0.15037838042245094
    phrase.count: 1
    phrase.text: 'minimal generating sets'
ic| phrase.chunks: [minimal generating sets]
ic| phrase.rank: 0.14740065982407313
    phrase.count: 1
    phrase.text: 'nonstrict inequations'
ic| phrase.chunks: [nonstrict inequations]
ic| phrase.rank: 0.13946027725597837
    phrase.count: 1
    phrase.text: 'strict inequations'
ic| phrase.chunks: [strict inequations]
ic| phrase.rank: 0.1195023546245721
    phrase.count: 1
    phrase.text: 'linear Diophantine equations'
ic| phrase.chunks: [linear Diophantine equations]
ic| phrase.rank: 0.11450088293222845
    phrase.count: 1
    phrase.text: 'natural numbers'
ic| phrase.chunks: [natural numbers]
ic| phrase.rank: 0.10780718173686318
    phrase.count: 3
    phrase.text: 'solutions'
ic| phrase.chunks: [solutions, solutions, solutions]
ic| phrase.rank: 0.10529828014583348
    phrase.count: 1
    phrase.text: 'linear constraints'
ic| phrase.chunks: [linear constraints]
ic| phrase.rank: 0.1036960590708142
    phrase.count: 1
    phrase.text: 'all the considered types systems'
ic| phrase.chunks: [all the considered types systems]
ic| phrase.rank: 0.08812713074893187
    phrase.count: 1
    phrase.text: 'a minimal supporting set'
ic| phrase.chunks: [a minimal supporting set]
ic| phrase.rank: 0.08444534702772151
    phrase.count: 1
    phrase.text: 'linear'
ic| phrase.chunks: [linear]
ic| phrase.rank: 0.08243620500315359
    phrase.count: 1
    phrase.text: 'a system'
ic| phrase.chunks: [a system]
ic| phrase.rank: 0.07944607954086784
    phrase.count: 1
    phrase.text: 'a minimal set'
ic| phrase.chunks: [a minimal set]
ic| phrase.rank: 0.0763527926213032
    phrase.count: 1
    phrase.text: 'algorithms'
ic| phrase.chunks: [algorithms]
ic| phrase.rank: 0.07593126037016427
    phrase.count: 1
    phrase.text: 'all types'
ic| phrase.chunks: [all types]
ic| phrase.rank: 0.07309361902551355
    phrase.count: 1
    phrase.text: 'Diophantine'
ic| phrase.chunks: [Diophantine]
ic| phrase.rank: 0.0702090100898443
    phrase.count: 1
    phrase.text: 'construction'
ic| phrase.chunks: [construction]
ic| phrase.rank: 0.05800111772673988
    phrase.count: 1
    phrase.text: 'the set'
ic| phrase.chunks: [the set]
ic| phrase.rank: 0.054251394765316464
    phrase.count: 1
    phrase.text: 'components'
ic| phrase.chunks: [components]
ic| phrase.rank: 0.04516904342912139
    phrase.count: 1
    phrase.text: 'Compatibility'
ic| phrase.chunks: [Compatibility]
ic| phrase.rank: 0.04516904342912139
    phrase.count: 1
    phrase.text: 'compatibility'
ic| phrase.chunks: [compatibility]
ic| phrase.rank: 0.04435648606848154
    phrase.count: 1
    phrase.text: 'the corresponding algorithms'
ic| phrase.chunks: [the corresponding algorithms]
ic| phrase.rank: 0.042273783712246285
    phrase.count: 1
    phrase.text: 'Criteria'
ic| phrase.chunks: [Criteria]
ic| phrase.rank: 0.01952542432474353
    phrase.count: 1
    phrase.text: 'These criteria'
ic| phrase.chunks: [These criteria]
```

--------------------------------

### Basic TextRank Pipeline Setup in spaCy

Source: https://context7.com/derwenai/pytextrank/llms.txt

Demonstrates how to load a spaCy model, add the PyTextRank component, process text, and access extracted phrases ranked by importance. Use this for standard phrase extraction.

```python
import spacy
import pytextrank

# Load a spaCy model
lp = spacy.load("en_core_web_sm")

# Add PyTextRank to the spaCy pipeline
lp.add_pipe("textrank")

# Example text for processing
text = """Compatibility of systems of linear constraints over the set of natural numbers.
Criteria of compatibility of a system of linear Diophantine equations, strict inequations,
and nonstrict inequations are considered. Upper bounds for components of a minimal set of
solutions and algorithms of construction of minimal generating sets of solutions for all
types of systems are given."""

# Process the document
doc = nlp(text)

# Access extracted phrases ranked by importance
for phrase in doc._.phrases:
    print(f"Phrase: {phrase.text}")
    print(f"Rank: {phrase.rank:.4f}, Count: {phrase.count}")
    print(f"Chunks: {phrase.chunks}")
    print("---")

# Output:
# Phrase: minimal set
# Rank: 0.1847, Count: 2
# Chunks: [minimal set, minimal set]
# ---
# Phrase: linear constraints
# Rank: 0.1234, Count: 1
# ..."
```

--------------------------------

### Extracting Noun Chunks for Phrase Analysis

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

This output shows an example of a noun chunk identified by the system, including its text, start and end positions, and length. This is part of the process to find minimal phrase representations.

```text
ic| token.lemma_: 'Diophantine'
    token.pos_: 'PROPN'
    node_id: 8
    rank: 0.03093250736456608
ic| phrase_rank: 0.08793819898736567
    chunk.text: 'Diophantine'
    chunk.start: 21
    chunk.end: 22
    chunk_len: 1
    counts[compound_key]: 1
```

--------------------------------

### Configure Pre-commit Hooks

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Installs and configures pre-commit hooks for local development. This ensures code quality checks are run automatically before commits.

```bash
pre-commit install
git config --local core.hooksPath .git/hooks/
```

--------------------------------

### Run spaCy Info Command

Source: https://github.com/derwenai/pytextrank/blob/main/wip/error.ipynb

Executes the spaCy command-line interface to display information about the installed spaCy version and its components in markdown format.

```bash
!python -m spacy info --markdown
```

--------------------------------

### Initialize Sentence Object

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Constructor for the Sentence data class. Requires start and end indices, sentence ID, phrases, and distance.

```python
__init__(start, end, sent_id, phrases, distance)
```

--------------------------------

### Load spaCy Language Model

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_summ.ipynb

Loads the English language model 'en_core_web_sm' for spaCy. Ensure the model is installed before running.

```python
import spacy
nlp = spacy.load("en_core_web_sm")
```

--------------------------------

### Initialize Sentence Boundaries

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_summ.ipynb

Creates a list of sentence boundaries, where each element contains the start and end index of a sentence and an empty set to store phrase IDs.

```python
sent_bounds = [ [s.start, s.end, set([])] for s in doc.sents ]
sent_bounds
```

--------------------------------

### Add User Bin to PATH (Unix-like)

Source: https://github.com/derwenai/pytextrank/blob/main/docs/setup.md

For Unix-like systems, add the user-level bin directory to your PATH environment variable if you installed JupyterLab with `pip install --user`.

```bash
export PATH="$HOME/.local/bin:$PATH"
```

--------------------------------

### Display spaCy Environment Information

Source: https://github.com/derwenai/pytextrank/blob/main/docs/faq.md

Run this command in your terminal to gather information about your spaCy installation, including versions and operating system details, which is helpful for troubleshooting.

```bash
python -m spacy info
```

--------------------------------

### Get Personalization for Personalized PageRank

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Get the node weights for initializing the use of the Personalized PageRank algorithm. Defaults to a no-op for the base TextRank algorithm.

```python
get_personalization()
```

--------------------------------

### Visualize Lemma Graph with Matplotlib

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

Use Matplotlib to draw the lemma graph. This requires a pre-existing lemma_graph and labels dictionary. Ensure matplotlib and networkx are installed.

```python
%matplotlib inline
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(9, 9))
pos = nx.spring_layout(lemma_graph)

nx.draw(lemma_graph, pos=pos, with_labels=False, font_weight="bold")
nx.draw_networkx_labels(lemma_graph, pos, labels);
```

--------------------------------

### Example Ranked Phrases Output

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

This output displays the top-ranked key phrases extracted by PyTextRank, including their counts and calculated ranks. It showcases phrases like 'minimal generating sets' and 'systems'.

```text
ic| phrase: 'minimal generating sets'
    count: 1
    rank: 0.17764305670256655
ic| phrase: 'systems', count: 4, rank: 0.15579774200552843
ic| phrase: 'linear diophantine equations'
    count: 1
    rank: 0.13635130800294415
ic| phrase: 'solutions', count: 3, rank: 0.12432459506251015
ic| phrase: 'mixed types', count: 1, rank: 0.12010505939797736
ic| phrase: 'nonstrict inequations'
```

--------------------------------

### Get Repository Version Information

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Retrieves Git repository version tag and commit hash for identifying the running version.

```python
get_repo_version()
```

--------------------------------

### Plotting Keyphrase Profile with Altair

Source: https://context7.com/derwenai/pytextrank/llms.txt

Generates an interactive Altair chart visualizing the document's keyphrase rank profile. Requires optional visualization dependencies (`pip install 'pytextrank[viz]'`). The chart is saved as an HTML file.

```python
import spacy
import pytextrank

# Install visualization dependencies: pip install 'pytextrank[viz]'
lp = spacy.load("en_core_web_sm")
lp.add_pipe("textrank")

text = """Artificial intelligence encompasses machine learning and deep learning.
Neural networks form the foundation of modern AI systems.
Natural language processing enables human-computer interaction.
Computer vision allows machines to interpret visual information."""

doc = nlp(text)
tr = doc._.textrank

try:
    # Generate interactive Altair chart
    chart = tr.plot_keyphrases()

    # Save as HTML for viewing in browser
    chart.save("keyphrase_profile.html")

    # Or display in Jupyter notebook
    # chart.display()

    print("Chart saved to keyphrase_profile.html")
except ImportError:
    print("Install visualization deps: pip install 'pytextrank[viz]'")
```

--------------------------------

### Analyze spaCy Pipeline Components

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Get a detailed overview of the spaCy pipeline's components, their configurations, and performance metrics using `analyze_pipes`.

```python
nlp.analyze_pipes(pretty=True)
```

--------------------------------

### Add PositionRank to spaCy Pipeline

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Load a spaCy model and add the 'positionrank' pipe to enable PositionRank functionality. This is the initial setup required before processing text.

```python
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("positionrank");
```

--------------------------------

### Add PyTextRank to spaCy Pipeline

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Import necessary libraries and add the 'textrank' component to the spaCy pipeline. This setup is for tutorial and development use; avoid modifying sys.path in production.

```python
# for use in tutorial and development; do not include this `sys.path` change in production:
import sys ; sys.path.insert(0, "../")
```

```python
import pytextrank
import spacy

lp = spacy.load("en_core_web_sm")
lp.add_pipe("textrank");
```

--------------------------------

### Plot Keyphrases Rank Profile

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Generates a plot of a document's keyphrases rank profile using Altair. Requires 'altair' and 'pandas' libraries to be installed.

```python
plot_keyphrases()
```

--------------------------------

### Add TextRank with Prefix Scrubber

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Configure PyTextRank to use the custom prefix scrubber. This setup processes a document and prints the top 10 phrases, demonstrating how the scrubber affects phrase normalization.

```python
nlp.add_pipe("textrank", config={ "stopwords": { "word": ["NOUN"] }, "scrubber": {"@misc": "prefix_scrubber"}})

doc = nlp(text)

for phrase in doc._.phrases[:10]:
    ic(phrase)
```

--------------------------------

### Package Documentation for Deployment

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Creates a tarball archive of the generated documentation site, suitable for deployment on a Flask/WSGI server.

```bash
tar cvzf ptr.tgz site/
```

--------------------------------

### Build Documentation

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Builds the project documentation using MkDocs and related tools. This process involves running shell scripts and the mkdocs build command.

```bash
./bin/nb_md.sh
./pkg_doc.py docs/ref.md
mkdocs build
```

--------------------------------

### Phrase.__init__

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Initializes a Phrase object.

```APIDOC
## Phrase.__init__

### Description
Initializes a Phrase object.

### Method
POST

### Endpoint
/derwenai/pytextrank

### Parameters
#### Request Body
- **text** (str) - Required - The text of the phrase.
- **chunks** (list) - Required - The chunks that make up the phrase.
- **count** (int) - Required - The count of the phrase.
- **rank** (float) - Required - The rank of the phrase.
```

--------------------------------

### Get Python Executable Path

Source: https://github.com/derwenai/pytextrank/blob/main/wip/error.ipynb

Retrieves the absolute path of the current Python executable. Useful for environment verification.

```python
sys.executable
```

--------------------------------

### TopicRank: __init__ Method

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Constructor for the factory used to instantiate PyTextRank pipeline components.

```APIDOC
### `__init__` method

```python
__init__(doc, edge_weight, pos_kept, token_lookback, scrubber, stopwords, threshold, method)
```

Constructor for a factory used to instantiate the PyTextRank pipeline components.

* `edge_weight` : `float`
default weight for an edge

* `pos_kept` : `typing.List[str]`
parts of speech tags to be kept; adjust this if strings representing the POS tags change

* `token_lookback` : `int`
the window for neighboring tokens – similar to a *skip gram*

* `scrubber` : `typing.Callable`
optional "scrubber" function to clean up punctuation from a token; if `None` then defaults to `pytextrank.default_scrubber`; when running, PyTextRank will throw a `FutureWarning` warning if the configuration uses a deprecated approach for a scrubber function

* `stopwords` : `typing.Dict[str, typing.List[str]]`
optional dictionary of `lemma: [pos]` items to define the *stop words*, where each item has a key as a lemmatized token and a value as a list of POS tags; may be a file name (string) or a [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html) for a JSON file; otherwise throws a `TypeError` exception

* `threshold` : `float`
threshold used in *TopicRank* candidate clustering; the original algorithm uses 0.25

* `method` : `str`
clustering method used in *TopicRank* candidate clustering: see [`scipy.cluster.hierarchy.linkage`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.linkage.html) for valid methods; the original algorithm uses "average"
```

--------------------------------

### Initialize NetworkX Graph

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

Imports the NetworkX library, which is used for graph manipulation.

```python
import networkx as nx
```

--------------------------------

### Detect Sentence Boundaries

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

Iterates through the sentences in a spaCy processed document and prints their start and end token indices.

```python
from icecream import ic

for sent in doc.sents:
    ic(sent.start, sent.end)
```

--------------------------------

### Initialization Method (Phrase)

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Initializes a Phrase object with text, chunks, count, and rank.

```python
__init__(text, chunks, count, rank)
```

--------------------------------

### Load Text and Initialize spaCy Pipeline

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Loads text from a file and initializes the spaCy NLP pipeline with PyTextRank. Ensure the 'nlp' object is already configured with the PyTextRank component.

```python
from icecream import ic
import pathlib

text = pathlib.Path("../dat/mih.txt").read_text()
text
```

--------------------------------

### Add PyTextRank to spaCy Pipeline

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_summ.ipynb

Integrates the 'textrank' component into the spaCy pipeline and processes the defined text. Requires 'pytextrank' to be installed.

```python
import pytextrank

nlp.add_pipe("textrank", last=True)
doc = nlp(text)
```

--------------------------------

### Render GraphViz DOT file using Python

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Import the `graphviz` library and use `Source.from_file` to load and represent the DOT file. This object can be rendered in environments like Jupyter notebooks.

```python
import graphviz as gv

gv.Source.from_file("lemma_graph.dot")
```

--------------------------------

### Verify GraphViz DOT file creation

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Use a shell command to list the created GraphViz DOT file, confirming its existence and size.

```bash
!ls -lth lemma_graph.dot
```

--------------------------------

### Load spaCy and Add TopicRank Pipe

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Initializes a spaCy model and adds the 'topicrank' component to the pipeline. This is the first step before processing text with TopicRank.

```python
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("topicrank");
```

--------------------------------

### Get Sentence Text from spaCy Doc

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Retrieves the text slice for a sentence from a spaCy Doc object. Requires the source document as input.

```python
text(doc)
```

--------------------------------

### Ignore Warnings in Jupyter

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_summ.ipynb

Suppresses warnings for cleaner output in Jupyter notebooks. No specific setup is required beyond importing the warnings module.

```python
import warnings
warnings.filterwarnings("ignore")
```

--------------------------------

### Token and Node Tracking in PyTextRank

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

Illustrates the step-by-step tracking of visited tokens and nodes during text processing. Useful for understanding the internal state of the algorithm.

```text
ic| list(range(len(visited_tokens) - 1, -1, -1)): [2, 1, 0]
ic| prev_token: 2, token.i - visited_tokens[prev_token]: 3
ic| node0: 17, node1: 16
ic| prev_token: 1, token.i - visited_tokens[prev_token]: 5
ic| token.i: 39
    token.text: 'minimal'
    token.lemma_: 'minimal'
    token.pos_: 'ADJ'
    visited_tokens: [33, 34, 36]
    visited_nodes: [14, 15, 16]
ic| visited_tokens: [33, 34, 36, 39], visited_nodes: [14, 15, 16, 17]
ic| list(range(len(visited_tokens) - 1, -1, -1)): [3, 2, 1, 0]
ic| prev_token: 3, token.i - visited_tokens[prev_token]: 1
ic| node0: 4, node1: 17
ic| prev_token: 2, token.i - visited_tokens[prev_token]: 4
ic| token.i: 40
    token.text: 'set'
    token.lemma_: 'set'
    token.pos_: 'NOUN'
    visited_tokens: [33, 34, 36, 39]
    visited_nodes: [14, 15, 16, 17]
ic| visited_tokens: [33, 34, 36, 39, 40]
    visited_nodes: [14, 15, 16, 17, 4]
ic| list(range(len(visited_tokens) - 1, -1, -1)): [4, 3, 2, 1, 0]
ic| prev_token: 4, token.i - visited_tokens[prev_token]: 2
ic| node0: 18, node1: 4
ic| prev_token: 3, token.i - visited_tokens[prev_token]: 3
ic| node0: 18, node1: 17
ic| prev_token: 2, token.i - visited_tokens[prev_token]: 6
ic| token.i: 42
    token.text: 'solutions'
    token.lemma_: 'solution'
    token.pos_: 'NOUN'
    visited_tokens: [33, 34, 36, 39, 40]
    visited_nodes: [14, 15, 16, 17, 4]
ic| visited_tokens: [33, 34, 36, 39, 40, 42]
    visited_nodes: [14, 15, 16, 17, 4, 18]
ic| list(range(len(visited_tokens) - 1, -1, -1)): [5, 4, 3, 2, 1, 0]
ic| prev_token: 5, token.i - visited_tokens[prev_token]: 2
ic| node0: 19, node1: 18
ic| prev_token: 4, token.i - visited_tokens[prev_token]: 4
ic| token.i: 44
    token.text: 'algorithms'
    token.lemma_: 'algorithm'
    token.pos_: 'NOUN'
    visited_tokens: [33, 34, 36, 39, 40, 42]
    visited_nodes: [14, 15, 16, 17, 4, 18]
ic| visited_tokens: [33, 34, 36, 39, 40, 42, 44]
    visited_nodes: [14, 15, 16, 17, 4, 18, 19]
ic| list(range(len(visited_tokens) - 1, -1, -1)): [6, 5, 4, 3, 2, 1, 0]
ic| prev_token: 6, token.i - visited_tokens[prev_token]: 2
ic| node0: 20, node1: 19
ic| prev_token: 5, token.i - visited_tokens[prev_token]: 4
ic| token.i: 46
    token.text: 'construction'
    token.lemma_: 'construction'
    token.pos_: 'NOUN'
    visited_tokens: [33, 34, 36, 39, 40, 42, 44]

```

--------------------------------

### Configure TextRank with Custom Parameters

Source: https://context7.com/derwenai/pytextrank/llms.txt

Shows how to customize PyTextRank's phrase extraction by setting parameters like edge weight, POS tags to keep, token lookback window, and custom stop words. Useful for fine-tuning results.

```python
import spacy
import pytextrank

nlp = spacy.load("en_core_web_sm")

# Configure with custom settings
nlp.add_pipe("textrank", config={
    "edge_weight": 1.0,           # Default weight for graph edges
    "pos_kept": ["ADJ", "NOUN", "PROPN", "VERB"],  # POS tags to include
    "token_lookback": 3,          # Window size for neighboring tokens
    "stopwords": {                # Custom stop words (lemma: [POS tags])
        "word": ["NOUN"],
        "system": ["NOUN"]
    }
})

text = """The word processing system handles document generation efficiently.
Each word in the system contributes to the overall word count."""

doc = nlp(text)

# Phrases with "word" and "system" filtered out
for phrase in doc._.phrases[:5]:
    print(f"{phrase.rank:.4f} - {phrase.text}")
```

--------------------------------

### Initialize TopicRank Constructor

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Constructor for the factory used to instantiate PyTextRank pipeline components. Adjust parameters like POS tags to keep, token lookback window, and clustering threshold as needed.

```python
__init__(doc, edge_weight, pos_kept, token_lookback, scrubber, stopwords, threshold, method)
```

--------------------------------

### Construct lemma graph from sentences

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

Initializes a NetworkX graph and iterates through sentences in a document to build a lemma graph. Requires a `doc` object and a `link_sentence` function.

```python
lemma_graph = nx.Graph()
seen_lemma = {}

for sent in doc.sents:
    link_sentence(doc, sent, lemma_graph, seen_lemma)
    #break # only test one sentence
```

--------------------------------

### Log Phrase Information

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

Logs detailed information about the processed phrase, including its calculated rank, original text, start and end positions, length, and current count.

```python
ic(phrase_rank, chunk.text, chunk.start, chunk.end, chunk_len, counts[compound_key])
```

--------------------------------

### Get Unit Vector for Top Phrases

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Constructs a unit vector representing top-ranked phrases in a spaCy Doc. Use this to compare sentences against the document's characteristic phrases.

```python
get_unit_vector(limit_phrases)
```

--------------------------------

### Creating a Label Dictionary for Graph Visualization

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

Code for creating a dictionary mapping numerical indices to lowercase lemma labels. This is a preparatory step for visualizing lemma graphs.

```python
labels = {}
keys = list(seen_lemma.keys())

for i in range(len(seen_lemma)):
    labels[i] = keys[i][0].lower()

labels
```

```text
Result:
{
0: 'compatibility',
1: 'system',

```

--------------------------------

### Configure PyTextRank with Custom Scrubber

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Load a spaCy model and add the 'textrank' pipeline component, configuring it with custom stopwords and the previously defined entity scrubber. Process text and iterate through the top phrases.

```python
nlp = spacy.load("en_core_web_sm")
lp.add_pipe("textrank", config={ "stopwords": { "word": ["NOUN"] }, "scrubber": {"@misc": "entity_scrubber"}})

doc = nlp(text)

for phrase in doc._.phrases[:10]:
    if phrase.text != "INELIGIBLE_PHRASE":
        ic(phrase)
```

--------------------------------

### Define a Prefix Scrubber Function

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

This function removes common prefixes like determiners and numbers from the beginning of a span before returning its lemma. It's useful for normalizing phrases that start with variations of common words.

```python
from spacy.tokens import Span
nlp = spacy.load("en_core_web_sm")


@spacy.registry.misc("prefix_scrubber")
def prefix_scrubber():
	def scrubber_func(span: Span) -> str:
		while len(span) > 1 and span[0].text in ("a", "the", "their", "every", "other", "two"):
			span = span[1:]
		return span.lemma_
	return scrubber_func
```

--------------------------------

### Initialize TopicRankFactory

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Constructor for the TopicRankFactory class, used to configure TopicRank parameters such as edge weight, token lookback, and similarity threshold.

```python
__init__(edge_weight=1.0, pos_kept=None, token_lookback=3, scrubber=None, stopwords=None, threshold=0.25, method="average")
```

--------------------------------

### Load spaCy and Add BiasedTextRank Pipe

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

Initializes a spaCy model and adds the 'biasedtextrank' component to the pipeline. This is required for using the Biased TextRank algorithm.

```python
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("biasedtextrank");
```

--------------------------------

### Sentence Distance Calculation for Custom Summarization

Source: https://context7.com/derwenai/pytextrank/llms.txt

Calculates the distance of each sentence from the document's unit vector of top phrases. This is useful for implementing custom summarization logic. It involves getting the unit vector of top phrases and then calculating sentence distances.

```python
import spacy
import pytextrank

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("textrank")

text = """Python is a versatile programming language.
It excels in data science and machine learning applications.
Web development with Python uses frameworks like Django and Flask.
Python's simple syntax makes it ideal for beginners.
Data scientists prefer Python for statistical analysis."""

doc = nlp(text)
tr = doc._.textrank

# Get unit vector of top phrases
unit_vector = tr.get_unit_vector(limit_phrases=10)
print("Unit vector (phrase coordinates):")
for elem in unit_vector[:5]:
    print(f"  {elem.phrase.text}: {elem.coord:.4f}")

# Calculate sentence distances
sent_distances = tr.calc_sent_dist(limit_phrases=10)
print("\nSentence distances from unit vector:")
for sent in sorted(sent_distances, key=lambda s: s.distance):
    text = doc[sent.start:sent.end].text.strip()
    print(f"  Distance {sent.distance:.4f}: {text[:60]}...")
```

--------------------------------

### Generate and Upload Coverage Report

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Generates a coverage report and uploads it to Codecov.io. Requires a Codecov.io access token stored in a file named .cc_token.

```bash
coverage report
bash <(curl -s https://codecov.io/bash) -t @.cc_token
```

--------------------------------

### Initialize BaseTextRankFactory

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Constructor for a factory used to instantiate the PyTextRank pipeline components. Configure edge weight, parts of speech to keep, token lookback window, and optional scrubber or stopwords.

```python
__init__(edge_weight=1.0, pos_kept=None, token_lookback=3, scrubber=None, stopwords=None)
```

--------------------------------

### TopicRank Algorithm for Keyphrase Clustering

Source: https://context7.com/derwenai/pytextrank/llms.txt

Implements TopicRank to cluster candidate keyphrases into topics using hierarchical agglomerative clustering. This reduces redundancy by grouping similar phrases. Configure clustering parameters like `threshold` and `method`.

```python
import spacy
import pytextrank

nlp = spacy.load("en_core_web_sm")

# TopicRank with clustering parameters
nlp.add_pipe("topicrank", config={
    "threshold": 0.25,   # Minimum overlap for clustering (25%)
    "method": "average"  # Clustering linkage method
})

text = """Machine learning algorithms process large datasets efficiently.
Deep learning neural networks have revolutionized computer vision.
Convolutional neural networks excel at image recognition tasks.
ML models require substantial training data and computational resources.
Deep neural networks can identify patterns in complex visual data."""

doc = nlp(text)

# Topics are clusters of related phrases
for phrase in doc._.phrases:
    print(f"Topic: {phrase.text}")
    print(f"Rank: {phrase.rank:.4f}, Members: {phrase.count}")
    # phrase.chunks contains all keyphrases in this topic cluster
    print(f"Cluster members: {[str(c) for c in phrase.chunks]}")
    print("---")
```

--------------------------------

### Initialize BaseTextRank

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Constructor for a TextRank object. Requires a spaCy Doc and configuration parameters like edge weight, POS tags to keep, token lookback, scrubber, and stopwords.

```python
__init__(doc, edge_weight, pos_kept, token_lookback, scrubber, stopwords)
```

--------------------------------

### Generate Conda-Forge Recipe

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Uses Grayskull to generate a Conda-Forge recipe for the PyTextRank package from its PyPI information.

```bash
grayskull pypi pytextrank
mv pytextrank/meta.yaml ./
```

--------------------------------

### Run Security Check with Bandit

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Analyzes Python code for common security vulnerabilities using Bandit.

```bash
bandit pytextrank/*.py
```

--------------------------------

### Launch JupyterLab from Virtual Environment

Source: https://github.com/derwenai/pytextrank/blob/main/docs/setup.md

Launch JupyterLab from within your activated virtual environment to avoid potential path issues with conda.

```bash
./venv/bin/jupyter-lab
```

--------------------------------

### Basic PyTextRank Usage with spaCy

Source: https://github.com/derwenai/pytextrank/blob/main/README.md

Load a spaCy model, add the 'textrank' pipeline, and process text to extract ranked phrases. This demonstrates the core functionality of PyTextRank.

```python
import spacy
import pytextrank

# example text
text = "Compatibility of systems of linear constraints over the set of natural numbers. Criteria of compatibility of a system of linear Diophantine equations, strict inequations, and nonstrict inequations are considered. Upper bounds for components of a minimal set of solutions and algorithms of construction of minimal generating sets of solutions for all types of systems are given. These criteria and the corresponding algorithms for constructing a minimal supporting set of solutions can be used in solving all the considered types systems and systems of mixed types."

# load a spaCy model, depending on language, scale, etc.
nlp = spacy.load("en_core_web_sm")

# add PyTextRank to the spaCy pipeline
nlp.add_pipe("textrank")
doc = nlp(text)

# examine the top-ranked phrases in the document
for phrase in doc._.phrases:
    print(phrase.text)
    print(phrase.rank, phrase.count)
    print(phrase.chunks)
```

--------------------------------

### Run Unit Tests with Coverage

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Executes unit tests using Pytest and measures test coverage with the Coverage tool. Ensure you are in the project root directory.

```bash
coverage run -m pytest tests
```

--------------------------------

### Collecting Lemmatized Tokens

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

Demonstrates the collection of lemmatized tokens and their associated indices. This output is crucial for building semantic graphs.

```text
ic(seen_lemma)
```

```text
ic| seen_lemma: {('Diophantine', 'PROPN'): {21},
                 ('Upper', 'PROPN'): {33},
                 ('algorithm', 'NOUN'): {66, 44},
                 ('bound', 'NOUN'): {34},
                 ('compatibility', 'NOUN'): {0, 15},
                 ('component', 'NOUN'): {36},
                 ('consider', 'VERB'): {82, 31},
                 ('constraint', 'NOUN'): {5},
                 ('construct', 'VERB'): {68},
                 ('construction', 'NOUN'): {46},
                 ('correspond', 'VERB'): {65},
                 ('criterion', 'NOUN'): {13, 62},
                 ('equation', 'NOUN'): {22},
                 ('generating', 'NOUN'): {49},
                 ('give', 'VERB'): {59},
                 ('inequation', 'NOUN'): {25, 29},
                 ('linear', 'ADJ'): {4, 20},
                 ('minimal', 'ADJ'): {48, 70, 39},
                 ('mixed', 'ADJ'): {88},
                 ('natural', 'ADJ'): {10},
                 ('nonstrict', 'ADJ'): {28},
                 ('number', 'NOUN'): {11},
                 ('set', 'NOUN'): {8, 40, 72, 50},
                 ('solution', 'NOUN'): {42, 52, 74},
                 ('solve', 'VERB'): {79},
                 ('strict', 'ADJ'): {24},
                 ('support', 'VERB'): {71},
                 ('system', 'NOUN'): {2, 18, 84, 86, 57},
                 ('type', 'NOUN'): {89, 83, 55},
                 ('use', 'VERB'): {77}}

```

```text
Result:
{('compatibility', 'NOUN'): {0, 15},
 ('system', 'NOUN'): {2, 18, 57, 84, 86},
 ('linear', 'ADJ'): {4, 20},
 ('constraint', 'NOUN'): {5},
 ('set', 'NOUN'): {8, 40, 50, 72},
 ('natural', 'ADJ'): {10},
 ('number', 'NOUN'): {11},
 ('criterion', 'NOUN'): {13, 62},
 ('Diophantine', 'PROPN'): {21},
 ('equation', 'NOUN'): {22},
 ('strict', 'ADJ'): {24},
 ('inequation', 'NOUN'): {25, 29},
 ('nonstrict', 'ADJ'): {28},
 ('consider', 'VERB'): {31, 82},
 ('Upper', 'PROPN'): {33},
 ('bound', 'NOUN'): {34},
 ('component', 'NOUN'): {36},
 ('minimal', 'ADJ'): {39, 48, 70},
 ('solution', 'NOUN'): {42, 52, 74},
 ('algorithm', 'NOUN'): {44, 66},
 ('construction', 'NOUN'): {46},
 ('generating', 'NOUN'): {49},
 ('type', 'NOUN'): {55, 83, 89},
 ('give', 'VERB'): {59},
 ('correspond', 'VERB'): {65},
 ('construct', 'VERB'): {68},
 ('support', 'VERB'): {71},
 ('use', 'VERB'): {77},
 ('solve', 'VERB'): {79},
 ('mixed', 'ADJ'): {88}}

```

--------------------------------

### Push Package Release to PyPI

Source: https://github.com/derwenai/pytextrank/blob/main/docs/build.md

Updates the release of the PyTextRank package on the Python Package Index (PyPI).

```bash
./bin/push_pypi.sh
```

--------------------------------

### Clone PyTextRank Repository

Source: https://github.com/derwenai/pytextrank/blob/main/docs/setup.md

Clone the PyTextRank Git repository to your local machine.

```bash
git clone https://github.com/DerwenAI/pytextrank.git
cd pytextrank
```

--------------------------------

### Initial Phrase Ranking

Source: https://github.com/derwenai/pytextrank/blob/main/examples/sample.ipynb

This code snippet shows the initial ranking of phrases from a text document before any stop words are applied. It reads text from a file and processes it using PyTextRank.

```python
text = pathlib.Path("../dat/gen.txt").read_text()
doc = nlp(text)

for phrase in doc._.phrases[:10]:
    ic(phrase)
```

--------------------------------

### Display Ranked Key Phrases

Source: https://github.com/derwenai/pytextrank/blob/main/examples/explain_algo.ipynb

This Python code iterates through the `min_phrases` dictionary, sorted by rank in descending order, and prints each phrase along with its count and rank. This presents the final ranked list of key phrases.

```python
for phrase, (rank, count) in sorted(min_phrases.items(), key=lambda x: x[1][0], reverse=True):
    ic(phrase, count, rank)
```

--------------------------------

### Initialize VectorElem Object

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Constructor for the VectorElem data class, representing an element in a document's unit vector.

```python
__init__(phrase, phrase_id, coord)
```

--------------------------------

### Custom Scrubber Functions for Phrase Cleaning

Source: https://context7.com/derwenai/pytextrank/llms.txt

Demonstrates using default and custom scrubber functions to clean phrase text. Reloads the pipeline with a custom scrubber that lowercases and removes punctuation.

```python
import spacy
import pytextrank
from pytextrank import default_scrubber, maniacal_scrubber

nlp = spacy.load("en_core_web_sm")

# Using default scrubber (removes apostrophes)
nlp.add_pipe("textrank", config={\"scrubber\": None})  # Uses default_scrubber

text = """The company's innovative approach to AI development
has attracted investors' attention worldwide."""

doc = nlp(text)
print("With default scrubber:")
for phrase in doc._.phrases[:3]:
    print(f"  {phrase.text}")

# Define a custom scrubber function
def custom_scrubber(span):
    """Custom scrubber that lowercases and removes punctuation."""
    import string
    text = span.text.lower()
    return text.translate(str.maketrans('', '', string.punctuation))

# Reload pipeline with custom scrubber
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("textrank", config={\"scrubber\": custom_scrubber})

doc = nlp(text)
print("\nWith custom scrubber:")
for phrase in doc._.phrases[:3]:
    print(f"  {phrase.text}")
```

--------------------------------

### Write Lemma Graph to DOT File

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Serializes the lemma graph into the DOT file format. Specify a path for the output file; defaults to 'graph.dot'.

```python
write_dot(path="graph.dot")
```

--------------------------------

### Lemma.label

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

Generates a simplified string representation of the Lemma object.

```APIDOC
## Lemma.label

### Description
Generates a more simplified string representation than `repr()`
provides.

### Method
GET

### Endpoint
/derwenai/pytextrank

### Returns
`str`
string representation
```

--------------------------------

### Add Custom spaCy Component

Source: https://github.com/derwenai/pytextrank/blob/main/wip/error.ipynb

Demonstrates how to add a custom component to a spaCy pipeline. Ensure the component is registered correctly using the Language.factory decorator.

```python
import inspect
import sys
import traceback

import spacy
import spacy.language

try:
    class MyComponent:
        def __init__(self, some_setting):
            print("got here")

    @spacy.language.Language.factory("my_component", default_config={"some_setting": True})
    def my_component(nlp, name, some_setting: bool):
        return MyComponent(some_setting=some_setting)

    nlp = spacy.blank("en")
    nlp.add_pipe("my_component", config={"some_setting": False})
except Exception:
    traceback.print_exc()
```

--------------------------------

### Set Attribute Method (Lemma)

Source: https://github.com/derwenai/pytextrank/blob/main/docs/ref.md

A standard method for setting attributes on a Lemma object.

```python
__setattr__(name, value)
```