### Install Development Dependencies Source: https://derwen.ai/docs/ptr/build Installs the necessary packages for local development and building the pytextrank library. Ensure you have Python 3 installed. ```bash python3 -m pip install -r requirements-dev.txt ``` -------------------------------- ### Install GraphViz Source: https://derwen.ai/docs/ptr/sample Installs the GraphViz Python library. This is a prerequisite for rendering DOT files programmatically. ```bash !pip install graphviz ``` -------------------------------- ### Install JupyterLab (pip) Source: https://derwen.ai/docs/ptr/setup Install JupyterLab using pip. If using --user, ensure the user-level bin directory is in your PATH. ```bash python3 -m pip install jupyterlab ``` -------------------------------- ### Install Dependencies (pip) Source: https://derwen.ai/docs/ptr/setup Install project dependencies using pip, including updating pip itself and downloading a spaCy model. ```bash python3 -m pip install -U pip python3 -m pip install -r requirements.txt python3 -m spacy download en_core_web_sm ``` -------------------------------- ### Install PyTextRank and spaCy Model Source: https://derwen.ai/docs/ptr/start Install the PyTextRank library from PyPI and download a small English spaCy model for basic text processing. ```bash python3 -m pip install pytextrank python3 -m spacy download en_core_web_sm ``` -------------------------------- ### Install Altair for Visualization Source: https://derwen.ai/docs/ptr/sample Installs the Altair library, used for creating interactive visualizations. This command is typically run in a shell environment. ```bash !pip install "altair" ``` -------------------------------- ### Install JupyterLab (conda) Source: https://derwen.ai/docs/ptr/setup Install JupyterLab using conda from the conda-forge channel. ```bash conda install -c conda-forge jupyterlab ``` -------------------------------- ### Display spaCy Installation Information Source: https://derwen.ai/docs/ptr/faq Run this command to gather information about your spaCy installation, including version and environment details. This is useful for debugging. ```bash python -m spacy info ``` -------------------------------- ### Preview Local Documentation Site Source: https://derwen.ai/docs/ptr/build Starts a local web server to preview the generated documentation microsite. Access the site by browsing to http://localhost:8000. ```bash ./bin/preview.py ``` -------------------------------- ### Install PyTextRank Dependencies from Git Source: https://derwen.ai/docs/ptr/start If working directly from the Git repository, install all necessary project dependencies using the requirements file. ```bash python3 -m pip install -r requirements.txt ``` -------------------------------- ### Create and Activate Virtual Environment (pip) Source: https://derwen.ai/docs/ptr/setup Create a Python virtual environment using venv and activate it before installing dependencies. ```bash python3 -m venv venv source venv/bin/activate ``` -------------------------------- ### PyTextRank Ranked Labels Output Example Source: https://derwen.ai/docs/ptr/explain_algo Example output showing the ranked labels and their corresponding scores as generated by the PyTextRank algorithm. This format is useful for understanding the most important terms or phrases identified in the text. ```text ic| labels[node_id]: 'system', rank: 0.09709174565608479 ic| labels[node_id]: 'set', rank: 0.07548767636963792 ic| labels[node_id]: 'minimal', rank: 0.06334806476862227 ic| labels[node_id]: 'solution', rank: 0.061826419749828485 ic| labels[node_id]: 'type', rank: 0.04816979699201436 ic| labels[node_id]: 'inequation', rank: 0.03969617593153302 ic| labels[node_id]: 'consider', rank: 0.0390375393827704 ic| labels[node_id]: 'linear', rank: 0.03658656272138432 ic| labels[node_id]: 'algorithm', rank: 0.03201021345587308 ic| labels[node_id]: 'construct', rank: 0.03165915652710971 ic| labels[node_id]: 'equation', rank: 0.031636552282656216 ic| labels[node_id]: 'component', rank: 0.031629446298645975 ic| labels[node_id]: 'diophantine', rank: 0.03093250736456608 ic| labels[node_id]: 'generating', rank: 0.029468555366439973 ic| labels[node_id]: 'support', rank: 0.029382686223731833 ic| labels[node_id]: 'correspond', rank: 0.026930354700088012 ic| labels[node_id]: 'compatibility', rank: 0.025190055141357165 ic| labels[node_id]: 'nonstrict', rank: 0.02513276636673567 ic| labels[node_id]: 'strict', rank: 0.0250439175297852 ic| labels[node_id]: 'upper', rank: 0.02428614673389346 ic| labels[node_id]: 'bound', rank: 0.02428614673389346 ic| labels[node_id]: 'construction', rank: 0.02404712231242087 ic| labels[node_id]: 'constraint', rank: 0.022947339381092696 ic| labels[node_id]: 'solve', rank: 0.01982332786706688 ic| labels[node_id]: 'criterion', rank: 0.019767433918161055 ic| labels[node_id]: 'use', rank: 0.019263849959229938 ic| labels[node_id]: 'natural', rank: 0.01884004785394172 ic| labels[node_id]: 'number', rank: 0.01884004785394172 ic| labels[node_id]: 'mixed', rank: 0.016743716826448173 ic| labels[node_id]: 'give', rank: 0.010894627731045426 ``` -------------------------------- ### Example Phrase Association Output Source: https://derwen.ai/docs/ptr/explain_summ Illustrates the output during the process of associating phrases with sentences, showing the phrase ID, text, rank, and the start and end indices of the chunk within the sentence boundaries. ```text ic| phrase_id: 0, p.text: 'mixed types', p.rank: 0.18359439311764025 ic| chunk.start: 88, chunk.end: 90 ``` -------------------------------- ### Load Example Text Source: https://derwen.ai/docs/ptr/sample Reads text content from a specified file path into a string variable. This is used as input for the NLP pipeline. ```python text = pathlib.Path("../dat/cfc.txt").read_text() text ``` -------------------------------- ### PositionRank Output Example Source: https://derwen.ai/docs/ptr/sample Example output from the PositionRank algorithm, showing extracted phrases with their text, constituent chunks, count, and rank. This illustrates the detailed information provided for each phrase. ```python ic| phrase: Phrase(text='deadline day', chunks=[deadline day, deadline day], count=2, rank=0.1671249044190727) ic| phrase: Phrase(text='Salomon Rondón', chunks=[Salomon Rondón, Salomon Rondón], count=2, rank=0.14836718147498046) ic| phrase: Phrase(text='Salomón Rondón', chunks=[Salomón Rondón, Salomón Rondón], count=2, rank=0.14169986334846618) ic| phrase: Phrase(text='Chelsea', chunks=[Chelsea, Chelsea, Chelsea, Chelsea], count=4, rank=0.13419811872859874) ic| phrase: Phrase(text='Rondón', chunks=[Rondón], count=1, rank=0.12722264594603172) ic| phrase: Phrase(text='a new centre', chunks=[a new centre], count=1, rank=0.09181159181129885) ic| phrase: Phrase(text='Giroud', chunks=[Giroud, Giroud], count=2, rank=0.0783201596831592) ic| phrase: Phrase(text='Olivier Giroud', chunks=[Olivier Giroud, Olivier Giroud], count=2, rank=0.07805316118093475) ic| phrase: Phrase(text='none', chunks=[none], count=1, rank=0.07503538984105931) ic| phrase: Phrase(text='their long search', chunks=[their long search], count=1, rank=0.07449683199895643) ``` -------------------------------- ### Example Output with Articles Scrubber Source: https://derwen.ai/docs/ptr/sample Shows the output after applying the articles_scrubber, where variations like 'sentences' and 'the sentences' are grouped together under a single phrase entry. ```python ic| phrase: Phrase(text='sentences', chunks=[sentences, the sentences], count=2, rank=0.14407118792073048) ic| phrase: Phrase(text='Mihalcea et al', chunks=[Mihalcea et al], count=1, rank=0.12123026637064825) ic| phrase: Phrase(text='Barrios et al', chunks=[Barrios et al], count=1, rank=0.11566772028535821) ic| phrase: Phrase(text='text summarization', chunks=[text summarization], count=1, rank=0.09524776232834677) ic| phrase: Phrase(text='ranking webpages', chunks=[ranking webpages], count=1, rank=0.08260919223940909) ic| phrase: Phrase(text='Okapi BM25 function', chunks=[Okapi BM25 function], count=1, rank=0.08125840606728206) ic| phrase: Phrase(text='gensim implements', chunks=[gensim implements], count=1, rank=0.08043607214961235) ic| phrase: Phrase(text='other sentence', chunks=[every other sentence], count=1, rank=0.07915141312258998) ic| phrase: Phrase(text='original TextRank', chunks=[original TextRank], count=1, ``` -------------------------------- ### Render GraphViz DOT file Source: https://derwen.ai/docs/ptr/sample Imports the graphviz library and loads a DOT file for rendering. Requires GraphViz to be installed on the system. ```python import graphviz as gv gv.Source.from_file("lemma_graph.dot") ``` -------------------------------- ### TextRank Output Example Source: https://derwen.ai/docs/ptr/sample Example output from the baseline TextRank algorithm, showcasing extracted phrases and their associated data. This output is compared against PositionRank to highlight differences in phrase selection and ranking. ```python ic| phrase: Phrase(text='Shanghai Shenhua striker Odion Ighalo', chunks=[Shanghai Shenhua striker Odion Ighalo, Shanghai Shenhua striker Odion Ighalo], count=2, rank=0.11863090071749424) ic| phrase: Phrase(text='none', chunks=[none], count=1, rank=0.09802416183300769) ic| phrase: Phrase(text='Moussa Dembele', chunks=[Moussa Dembele, Moussa Dembele], count=2, rank=0.09341044332809736) ic| phrase: Phrase(text='deadline day', chunks=[deadline day, deadline day], count=2, rank=0.09046182507994752) ic| phrase: Phrase(text='Dries Mertens', chunks=[Dries Mertens, Dries Mertens], count=2, rank=0.08919649435994934) ic| phrase: Phrase(text='Edinson Cavani', chunks=[Edinson Cavani], count=1, rank=0.08418633972470349) ic| phrase: Phrase(text='Salomon Rondón', chunks=[Salomon Rondón, Salomon Rondón], count=2, rank=0.08228367707127111) ic| phrase: Phrase(text='Salomón Rondón', chunks=[Salomón Rondón, Salomón Rondón], count=2, rank=0.08228367707127111) ic| phrase: Phrase(text='Rondón', chunks=[Rondón], count=1, rank=0.0750732870664833) ic| phrase: Phrase(text='Dalian Yifang', chunks=[Dalian Yifang, Dalian Yifang], count=2, rank=0.06681675615287698) ``` -------------------------------- ### Configure Pre-commit Hooks Source: https://derwen.ai/docs/ptr/build Sets up pre-commit hooks for automated code quality checks. This involves installing the hooks and configuring Git to use them. ```bash pre-commit install git config --local core.hooksPath .git/hooks/ ``` -------------------------------- ### Load spaCy and Add TopicRank Pipe Source: https://derwen.ai/docs/ptr/sample Initializes a spaCy model and adds the TopicRank component to the pipeline. Ensure the 'topicrank' component is installed. ```python nlp = spacy.load("en_core_web_sm") lp.add_pipe("topicrank") ``` -------------------------------- ### Example Output with Prefix Scrubber Source: https://derwen.ai/docs/ptr/sample Illustrates how the prefix scrubber groups variations of 'sentence' by their lemma, such as 'every sentence' and 'the two sentences', under a single phrase entry. ```python ic| phrase: Phrase(text='sentence', chunks=[sentences, every sentence, every other sentence, the two sentences, two sentences, the sentences], count=6, rank=0.14407118792073048) ic| phrase: Phrase(text='Mihalcea et al', chunks=[Mihalcea et al], count=1, rank=0.12123026637064825) ic| phrase: Phrase(text='Barrios et al', chunks=[Barrios et al], count=1, rank=0.11566772028535821) ic| phrase: Phrase(text='text summarization', chunks=[text summarization], count=1, rank=0.09524776232834677) ic| phrase: Phrase(text='rank webpage', chunks=[ranking webpages], count=1, rank=0.08260919223940909) ic| phrase: Phrase(text='Okapi BM25 function', chunks=[Okapi BM25 function], count=1, rank=0.08125840606728206) ic| phrase: Phrase(text='gensim implement', chunks=[gensim implements], count=1, rank=0.08043607214961235) ic| phrase: Phrase(text='original TextRank', chunks=[original TextRank], count=1, rank=0.07013026654397199) ic| phrase: Phrase(text='TextRank', chunks=[TextRank, TextRank, TextRank, TextRank], count=4, rank=0.06686718957926076) ic| phrase: Phrase(text='Olavur Mortensen', chunks=[Olavur Mortensen, Olavur Mortensen], count=2, rank=0.06548020385220721) ``` -------------------------------- ### Initialize Lemma Graph Source: https://derwen.ai/docs/ptr/explain_algo Initializes a NetworkX graph to represent the lemma graph and a dictionary to track seen lemmas. This is the starting point for building the graph structure. ```python lemma_graph = nx.Graph() seen_lemma = {} ``` -------------------------------- ### Display PyTextRank Phrase Results Source: https://derwen.ai/docs/ptr/explain_summ This is an example output showing the ranked phrases, their counts, text, and chunks as extracted by PyTextRank. ```text ic| p.rank: 0.18359439311764025, p.count: 1, p.text: 'mixed types' ic| p.chunks: [mixed types] ic| p.rank: 0.17847961931078207, p.count: 3, p.text: 'systems' ic| p.chunks: [systems, systems, systems] ic| p.rank: 0.15037838042245094 p.count: 1 p.text: 'minimal generating sets' ic| p.chunks: [minimal generating sets] ic| p.rank: 0.14740065982407316 p.count: 1 p.text: 'nonstrict inequations' ic| p.chunks: [nonstrict inequations] ic| p.rank: 0.13946027725597837 p.count: 1 p.text: 'strict inequations' ic| p.chunks: [strict inequations] ic| p.rank: 0.1195023546245721 p.count: 1 p.text: 'linear Diophantine equations' ic| p.chunks: [linear Diophantine equations] ic| p.rank: 0.11450088293222845, p.count: 1, p.text: 'natural numbers' ic| p.chunks: [natural numbers] ic| p.rank: 0.1078071817368632, p.count: 3, p.text: 'solutions' ic| p.chunks: [solutions, solutions, solutions] ic| p.rank: 0.10529828014583348 p.count: 1 p.text: 'linear constraints' ic| p.chunks: [linear constraints] ic| p.rank: 0.10369605907081418 p.count: 1 p.text: 'all the considered types systems' ic| p.chunks: [all the considered types systems] ic| p.rank: 0.08812713074893187 p.count: 1 p.text: 'a minimal supporting set' ic| p.chunks: [a minimal supporting set] ic| p.rank: 0.08243620500315357, p.count: 1, p.text: 'a system' ic| p.chunks: [a system] ic| p.rank: 0.07944607954086784, p.count: 1, p.text: 'a minimal set' ic| p.chunks: [a minimal set] ic| p.rank: 0.0763527926213032, p.count: 1, p.text: 'algorithms' ic| p.chunks: [algorithms] ic| p.rank: 0.07593126037016427, p.count: 1, p.text: 'all types' ic| p.chunks: [all types] ic| p.rank: 0.07309361902551356, p.count: 1, p.text: 'Diophantine' ic| p.chunks: [Diophantine] ic| p.rank: 0.0702090100898443, p.count: 1, p.text: 'construction' ic| p.chunks: [construction] ic| p.rank: 0.060225391238828516, p.count: 1, p.text: 'Upper bounds' ic| p.chunks: [Upper bounds] ic| p.rank: 0.05800111772673988, p.count: 1, p.text: 'the set' ic| p.chunks: [the set] ic| p.rank: 0.05425139476531647, p.count: 1, p.text: 'components' ic| p.chunks: [components] ic| p.rank: 0.04516904342912139, p.count: 1, p.text: 'Compatibility' ic| p.chunks: [Compatibility] ic| p.rank: 0.04516904342912139, p.count: 1, p.text: 'compatibility' ic| p.chunks: [compatibility] ic| p.rank: 0.04435648606848154 p.count: 1 p.text: 'the corresponding algorithms' ic| p.chunks: [the corresponding algorithms] ic| p.rank: 0.042273783712246285, p.count: 1, p.text: 'Criteria' ic| p.chunks: [Criteria] ic| p.rank: 0.01952542432474353, p.count: 1, p.text: 'These criteria' ic| p.chunks: [These criteria] ``` -------------------------------- ### Sentence Class: Initialization Source: https://derwen.ai/docs/ptr/ref Initializes a Sentence object with start, end, sentence ID, phrases, and distance. ```python __init__(start, end, sent_id, phrases, distance) ``` -------------------------------- ### Example of Ranked Phrases Output Source: https://derwen.ai/docs/ptr/explain_algo Shows a sample of the ranked phrases output from PyTextRank, including the phrase text, its frequency count, and its calculated rank. ```text ic| phrase: 'minimal generating sets' count: 1 rank: 0.17764305670256655 ic| phrase: 'systems', count: 4, rank: 0.15579774200552843 ic| phrase: 'linear diophantine equations' count: 1 rank: 0.13635130800294415 ic| phrase: 'solutions', count: 3, rank: 0.12432459506251015 ic| phrase: 'mixed types', count: 1, rank: 0.12010505939797736 ic| phrase: 'nonstrict inequations' count: 1 rank: 0.12002679543267614 ic| phrase: 'strict inequations', count: 1, rank: 0.11994451815672316 ic| phrase: 'linear constraints', count: 1, rank: 0.11502067650110856 ic| phrase: 'a minimal supporting set' count: 1 rank: 0.10481265770639074 ic| phrase: 'upper bounds', count: 1, rank: 0.10389342131646997 ic| phrase: 'a minimal set', count: 1, rank: 0.09315167105930662 ic| phrase: 'natural numbers', count: 1, rank: 0.09150603587606598 ic| phrase: 'algorithms', count: 1, rank: 0.08945699169974514 ic| phrase: 'components', count: 1, rank: 0.08892334662315343 ic| phrase: 'diophantine', count: 1, rank: 0.08793819898736567 ic| phrase: 'all the considered types systems' count: 1 rank: 0.0811302044403381 ic| phrase: 'compatibility', count: 2, rank: 0.07935687610622845 ic| phrase: 'construction', count: 1, rank: 0.07753567293901058 ic| phrase: 'criteria', count: 2, rank: 0.07029835332026109 ic| phrase: 'the set', count: 1, rank: 0.06345084244027568 ic| phrase: 'the corresponding algorithms' count: 1 rank: 0.0606941966727262 ic| phrase: 'all types', count: 1, rank: 0.050685854432712285 ``` -------------------------------- ### Example of a Single Noun Chunk Source: https://derwen.ai/docs/ptr/explain_algo Illustrates a single noun chunk identified by the algorithm, showing its lemma, part-of-speech, node ID, and rank. ```text ic| token.lemma_: 'Diophantine' token.pos_: 'PROPN' node_id: 8 rank: 0.03093250736456608 ic| phrase_rank: 0.08793819898736567 chunk.text: 'Diophantine' chunk.start: 21 chunk.end: 22 chunk_len: 1 counts[compound_key]: 1 ``` -------------------------------- ### Add PyTextRank to spaCy Pipeline Source: https://derwen.ai/docs/ptr/sample Import necessary libraries and add the PyTextRank component to your spaCy pipeline. This is the initial setup required to use PyTextRank. ```python import pytextrank import spacy lp = spacy.load("en_core_web_sm") lp.add_pipe("textrank"); ``` -------------------------------- ### Display Initial Sentence Boundaries Source: https://derwen.ai/docs/ptr/explain_summ Shows the initial structure of sentence boundaries, where each sentence is represented by its start and end token indices and an empty set for phrase vectors. ```text [[0, 13, set()], [13, 33, set()], [33, 61, set()], [61, 91, set()]] ``` -------------------------------- ### Add User Bin to PATH (Unix) Source: https://derwen.ai/docs/ptr/setup Add the user-level bin directory to your PATH environment variable for Unix-like systems when using pip install --user. ```bash export PATH="$HOME/.local/bin:$PATH" ``` -------------------------------- ### Get Personalization for Personalized PageRank Source: https://derwen.ai/docs/ptr/ref Retrieves the node weights for initializing the Personalized PageRank algorithm. Defaults to a no-op for the base TextRank algorithm. ```python get_personalization() ``` -------------------------------- ### PyTextRank Debug Output Example Source: https://derwen.ai/docs/ptr/explain_algo This snippet shows a detailed, line-by-line debug output from the PyTextRank algorithm during graph construction. It logs token information, visited nodes, and edge connections, useful for understanding the algorithm's internal state. ```python ic| visited_tokens: [], visited_nodes: [] ic| list(range(len(visited_tokens) - 1, -1, -1)): [] ic| token.i: 0 token.text: 'Compatibility' token.lemma_: 'compatibility' token.pos_: 'NOUN' visited_tokens: [] visited_nodes: [] ic| visited_tokens: [0], visited_nodes: [0] ic| list(range(len(visited_tokens) - 1, -1, -1)): [0] ic| prev_token: 0, token.i - visited_tokens[prev_token]: 2 ic| node0: 1, node1: 0 ic| token.i: 2 token.text: 'systems' token.lemma_: 'system' token.pos_: 'NOUN' visited_tokens: [0] visited_nodes: [0] ic| visited_tokens: [0, 2], visited_nodes: [0, 1] ic| list(range(len(visited_tokens) - 1, -1, -1)): [1, 0] ic| prev_token: 1, token.i - visited_tokens[prev_token]: 2 ic| node0: 2, node1: 1 ic| prev_token: 0, token.i - visited_tokens[prev_token]: 4 ic| token.i: 4 token.text: 'linear' token.lemma_: 'linear' token.pos_: 'ADJ' visited_tokens: [0, 2] visited_nodes: [0, 1] ic| visited_tokens: [0, 2, 4], visited_nodes: [0, 1, 2] ic| list(range(len(visited_tokens) - 1, -1, -1)): [2, 1, 0] ic| prev_token: 2, token.i - visited_tokens[prev_token]: 1 ic| node0: 3, node1: 2 ic| prev_token: 1, token.i - visited_tokens[prev_token]: 3 ic| node0: 3, node1: 1 ic| prev_token: 0, token.i - visited_tokens[prev_token]: 5 ic| token.i: 5 token.text: 'constraints' token.lemma_: 'constraint' token.pos_: 'NOUN' visited_tokens: [0, 2, 4] visited_nodes: [0, 1, 2] ic| visited_tokens: [0, 2, 4, 5], visited_nodes: [0, 1, 2, 3] ic| list(range(len(visited_tokens) - 1, -1, -1)): [3, 2, 1, 0] ic| prev_token: 3, token.i - visited_tokens[prev_token]: 3 ic| node0: 4, node1: 3 ic| prev_token: 2, token.i - visited_tokens[prev_token]: 4 ic| token.i: 8 token.text: 'set' token.lemma_: 'set' token.pos_: 'NOUN' visited_tokens: [0, 2, 4, 5] visited_nodes: [0, 1, 2, 3] ic| visited_tokens: [0, 2, 4, 5, 8], visited_nodes: [0, 1, 2, 3, 4] ic| list(range(len(visited_tokens) - 1, -1, -1)): [4, 3, 2, 1, 0] ic| prev_token: 4, token.i - visited_tokens[prev_token]: 2 ic| node0: 5, node1: 4 ic| prev_token: 3, token.i - visited_tokens[prev_token]: 5 ic| token.i: 10 token.text: 'natural' token.lemma_: 'natural' token.pos_: 'ADJ' visited_tokens: [0, 2, 4, 5, 8] visited_nodes: [0, 1, 2, 3, 4] ic| visited_tokens: [0, 2, 4, 5, 8, 10] visited_nodes: [0, 1, 2, 3, 4, 5] ic| list(range(len(visited_tokens) - 1, -1, -1)): [5, 4, 3, 2, 1, 0] ic| prev_token: 5, token.i - visited_tokens[prev_token]: 1 ic| node0: 6, node1: 5 ic| prev_token: 4, token.i - visited_tokens[prev_token]: 3 ic| node0: 6, node1: 4 ic| prev_token: 3, token.i - visited_tokens[prev_token]: 6 ic| token.i: 11 token.text: 'numbers' token.lemma_: 'number' token.pos_: 'NOUN' visited_tokens: [0, 2, 4, 5, 8, 10] visited_nodes: [0, 1, 2, 3, 4, 5] ic| visited_tokens: [], visited_nodes: [] ic| list(range(len(visited_tokens) - 1, -1, -1)): [] ic| token.i: 13 token.text: 'Criteria' token.lemma_: 'criterion' token.pos_: 'NOUN' visited_tokens: [] visited_nodes: [] ic| visited_tokens: [13], visited_nodes: [7] ic| list(range(len(visited_tokens) - 1, -1, -1)): [0] ic| prev_token: 0, token.i - visited_tokens[prev_token]: 2 ic| node0: 0, node1: 7 ic| token.i: 15 token.text: 'compatibility' token.lemma_: 'compatibility' token.pos_: 'NOUN' visited_tokens: [13] visited_nodes: [7] ic| visited_tokens: [13, 15], visited_nodes: [7, 0] ic| list(range(len(visited_tokens) - 1, -1, -1)): [1, 0] ic| prev_token: 1, token.i - visited_tokens[prev_token]: 3 ic| node0: 1, node1: 0 ic| prev_token: 0, token.i - visited_tokens[prev_token]: 5 ic| token.i: 18 token.text: 'system' token.lemma_: 'system' token.pos_: 'NOUN' visited_tokens: [13, 15] visited_nodes: [7, 0] ic| visited_tokens: [13, 15, 18], visited_nodes: [7, 0, 1] ic| list(range(len(visited_tokens) - 1, -1, -1)): [2, 1, 0] ic| prev_token: 2, token.i - visited_tokens[prev_token]: 2 ic| node0: 2, node1: 1 ic| prev_token: 1, token.i - visited_tokens[prev_token]: 5 ic| token.i: 20 token.text: 'linear' token.lemma_: 'linear' token.pos_: 'ADJ' visited_tokens: [13, 15, 18] visited_nodes: [7, 0, 1] ic| visited_tokens: [13, 15, 18, 20], visited_nodes: [7, 0, 1, 2] ic| list(range(len(visited_tokens) - 1, -1, -1)): [3, 2, 1, 0] ic| prev_token: 3, token.i - visited_tokens[prev_token]: 1 ``` -------------------------------- ### BaseTextRank.get_personalization Source: https://derwen.ai/docs/ptr/ref Get the node weights for initializing the use of the Personalized PageRank algorithm. Defaults to a no-op for the base TextRank algorithm. ```APIDOC ## BaseTextRank.get_personalization ### Description Get the _node weights_ for initializing the use of the _Personalized PageRank_ algorithm. Defaults to a no-op for the base _TextRank_ algorithm. ### Returns - `typing.Union[typing.Dict[pytextrank.base.Lemma, float], NoneType]` - `None` ``` -------------------------------- ### Load spaCy Language Model Source: https://derwen.ai/docs/ptr/explain_algo Loads the English language model for spaCy. Ensure the model is installed (`python -m spacy download en_core_web_sm`). ```python import spacy lp = spacy.load("en_core_web_sm") ``` -------------------------------- ### Get Repository Version Function Source: https://derwen.ai/docs/ptr/ref Retrieves Git repository information to identify the version or commit hash of the running production code. ```python get_repo_version() ``` -------------------------------- ### plot_keyphrases Source: https://derwen.ai/docs/ptr/ref Generates a plot visualizing the document's keyphrase rank profile using Altair. Requires Altair and Pandas libraries to be installed. ```APIDOC ## plot_keyphrases ### Description Plot a document's keyphrases rank profile using `altair.Chart` Throws an `ImportError` if the `altair` and `pandas` libraries are not installed. ### Returns - typing.Any - the `altair` chart being rendered ``` -------------------------------- ### Plot Keyphrases with Altair Source: https://derwen.ai/docs/ptr/sample Generates an interactive plot of keyphrases from the document's textrank object using Altair. This requires the Altair library to be installed. ```python tr = doc._.textrank tr.plot_keyphrases() ``` -------------------------------- ### Write GraphViz DOT file for Lemma Graph Source: https://derwen.ai/docs/ptr/sample Exports the lemma graph generated by PyTextRank to a GraphViz DOT file. Ensure GraphViz is installed to render the output. ```python tr = doc._.textrank tr.write_dot(path="lemma_graph.dot") ``` -------------------------------- ### Plot Keyphrase Rank Profile Source: https://derwen.ai/docs/ptr/ref Generates a plot visualizing the rank profile of a document's keyphrases using Altair. Requires 'altair' and 'pandas' to be installed. ```python plot_keyphrases() ``` -------------------------------- ### Build Documentation Source: https://derwen.ai/docs/ptr/build Builds the project's online documentation using MkDocs and related tools. This involves running a script to process notebooks and then building the static site. ```bash ./bin/nb_md.sh ./pkg_doc.py docs/ref.md mkdocs build ``` -------------------------------- ### Package Documentation Microsite Source: https://derwen.ai/docs/ptr/build Creates a tarball archive of the generated documentation site, suitable for deployment on a Flask/WSGI server. ```bash tar cvzf ptr.tgz site/ ``` -------------------------------- ### Sentence Class: Get Text Slice Source: https://derwen.ai/docs/ptr/ref Retrieves the text slice from a spaCy Doc object corresponding to this sentence. ```python text(doc) ``` -------------------------------- ### TopicRankFactory.__init__ Source: https://derwen.ai/docs/ptr/ref Constructor for the TopicRankFactory class, initializing parameters for creating TopicRank pipeline components. ```APIDOC ## TopicRankFactory.__init__ ### Description Constructor for the factory class. ### Parameters #### Path Parameters - **edge_weight** (float) - Optional - **pos_kept** (typing.Optional[typing.List[str]]) - Optional - **token_lookback** (int) - Optional - **scrubber** (typing.Optional[typing.Callable]) - Optional - **stopwords** (typing.Optional[typing.Set[str]]) - Optional - **threshold** (float) - Optional - **method** (str) - Optional ``` -------------------------------- ### Launch JupyterLab from Virtual Environment Source: https://derwen.ai/docs/ptr/setup Launch JupyterLab using the specific executable within the virtual environment to avoid path issues. ```bash ./venv/bin/jupyter-lab ``` -------------------------------- ### Create and Activate Conda Environment Source: https://derwen.ai/docs/ptr/setup Create a new conda environment from an environment file and activate it. ```bash conda env create -f environment.yml conda activate pytextrank ``` -------------------------------- ### Basic PyTextRank Usage with spaCy Source: https://derwen.ai/docs/ptr/start Demonstrates loading a spaCy model, adding the PyTextRank pipeline component, processing text, and iterating through the top-ranked phrases with their scores and associated chunks. ```python import spacy import pytextrank # example text text = "Compatibility of systems of linear constraints over the set of natural numbers. Criteria of compatibility of a system of linear Diophantine equations, strict inequations, and nonstrict inequations are considered. Upper bounds for components of a minimal set of solutions and algorithms of construction of minimal generating sets of solutions for all types of systems are given. These criteria and the corresponding algorithms for constructing a minimal supporting set of solutions can be used in solving all the considered types systems and systems of mixed types." # load a spaCy model, depending on language, scale, etc. nlp = spacy.load("en_core_web_sm") # add PyTextRank to the spaCy pipeline nlp.add_pipe("textrank") doc = nlp(text) # examine the top-ranked phrases in the document for phrase in doc._.phrases: print(phrase.text) print(phrase.rank, phrase.count) print(phrase.chunks) ``` -------------------------------- ### BaseTextRank.__init__ Source: https://derwen.ai/docs/ptr/ref Constructor for a TextRank object. ```APIDOC ## BaseTextRank.__init__ ### Description Constructor for a `TextRank` object. ### Parameters - **doc** (spacy.tokens.doc.Doc) - a document container, providing the annotations produced by earlier stages of the `spaCy` pipeline - **edge_weight** (float) - default weight for an edge - **pos_kept** (typing.List[str]) - parts of speech tags to be kept; adjust this if strings representing the POS tags change - **token_lookback** (int) - the window for neighboring tokens – similar to a _skip gram_ - **scrubber** (typing.Callable) - optional "scrubber" function to clean up punctuation from a token - **stopwords** (typing.Dict[str, typing.List[str]]) - optional dictionary of `lemma: [pos]` items to define the _stop words_ , where each item has a key as a lemmatized token and a value as a list of POS tags ``` -------------------------------- ### BaseTextRankFactory.__init__ Source: https://derwen.ai/docs/ptr/ref Constructor for a factory used to instantiate the PyTextRank pipeline components. ```APIDOC ## BaseTextRankFactory.__init__ ### Description Constructor for a factory used to instantiate the PyTextRank pipeline components. ### Parameters - **edge_weight** (float) - default weight for an edge - **pos_kept** (typing.List[str]) - parts of speech tags to be kept; adjust this if strings representing the POS tags change - **token_lookback** (int) - the window for neighboring tokens – similar to a _skip gram_ - **scrubber** (typing.Union[typing.Callable, NoneType]) - optional "scrubber" function to clean up punctuation from a token; if `None` then defaults to `pytextrank.default_scrubber`; when running, PyTextRank will throw a `FutureWarning` warning if the configuration uses a deprecated approach for a scrubber function - **stopwords** (typing.Union[str, pathlib.Path, typing.Dict[str, typing.List[str]], NoneType]) - optional dictionary of `lemma: [pos]` items to define the _stop words_ , where each item has a key as a lemmatized token and a value as a list of POS tags; may be a file name (string) or a `pathlib.Path` for a JSON file; otherwise throws a `TypeError` exception ``` -------------------------------- ### Initialize TopicRankFactory Source: https://derwen.ai/docs/ptr/ref Constructor for the TopicRank factory class. Allows configuration of edge weights, token lookback, and similarity thresholds. ```python __init__(edge_weight=1.0, pos_kept=None, token_lookback=3, scrubber=None, stopwords=None, threshold=0.25, method="average") ``` -------------------------------- ### Generate Conda-Forge Recipe Source: https://derwen.ai/docs/ptr/build Uses grayskull to generate a meta.yaml file for a conda-forge recipe based on the pytextrank PyPI package. The generated file is moved to the current directory. ```bash grayskull pypi pytextrank mv pytextrank/meta.yaml ./ ``` -------------------------------- ### BaseTextRankFactory.__call__ Source: https://derwen.ai/docs/ptr/ref Set the extension attributes on a spaCy Doc document to create a pipeline component for BaseTextRank as a stateful component, invoked when the document gets processed. ```APIDOC ## BaseTextRankFactory.__call__ ### Description Set the extension attributes on a spaCy `Doc` document to create a _pipeline component_ for `BaseTextRank` as a stateful component, invoked when the document gets processed. See: https://spacy.io/usage/processing-pipelines#pipelines ### Parameters - **doc** (spacy.tokens.doc.Doc) - a document container, providing the annotations produced by earlier stages of the `spaCy` pipeline ``` -------------------------------- ### Download spaCy Model (conda) Source: https://derwen.ai/docs/ptr/setup Download the en_core_web_sm model for spaCy within the activated conda environment. ```bash python3 -m spacy download en_core_web_sm ``` -------------------------------- ### Load Text from File Source: https://derwen.ai/docs/ptr/sample Loads text content from a specified file path using pathlib. Ensure the file exists at the given path. ```python from icecream import ic import pathlib text = pathlib.Path("../dat/mih.txt").read_text() text ``` -------------------------------- ### Construct Sentence Boundaries Source: https://derwen.ai/docs/ptr/explain_summ Initializes a list to store sentence boundaries and an empty set for phrase vectors for each sentence in the document. ```python sent_bounds = [ [s.start, s.end, set([])] for s in doc.sents ] sent_bounds ``` -------------------------------- ### Identify Sentence Boundaries with spaCy Source: https://derwen.ai/docs/ptr/explain_algo Iterates through sentences in a spaCy `Doc` object and prints their start and end token indices. Requires the `icecream` library for debugging output. ```python from icecream import ic for sent in doc.sents: ic(sent.start, sent.end) ``` -------------------------------- ### Load spaCy and Add Biased TextRank Pipe Source: https://derwen.ai/docs/ptr/sample Initializes a spaCy model and adds the Biased TextRank component to the pipeline. This algorithm allows for focusing on specific terms. ```python nlp = spacy.load("en_core_web_sm") lp.add_pipe("biasedtextrank") ``` -------------------------------- ### Add TextRank to spaCy Pipeline Source: https://derwen.ai/docs/ptr/sample Load a spaCy model and add the 'textrank' component to the pipeline for baseline phrase extraction. This setup is similar to PositionRank but uses the standard TextRank algorithm. ```python nlp = spacy.load("en_core_web_sm") nlp.add_pipe("textrank") ``` -------------------------------- ### Define Custom Entity Scrubber Function Source: https://derwen.ai/docs/ptr/sample This Python function, registered with spaCy's registry, checks if the first token of a span is a named entity. If it is, the phrase is marked as 'INELIGIBLE_PHRASE'; otherwise, the original text of the span is returned. Use this to filter out phrases that start with named entities. ```python @spacy.registry.misc("entity_scrubber") def articles_scrubber(): def scrubber_func(span: Span) -> str: if span[0].ent_type_: # ignore named entities return "INELIGIBLE_PHRASE" return span.text return scrubber_func ``` -------------------------------- ### Initialize BaseTextRankFactory Source: https://derwen.ai/docs/ptr/ref Constructor for a factory used to instantiate the PyTextRank pipeline components. Adjust pos_kept if POS tag strings change. The token_lookback defines the window for neighboring tokens. ```python __init__(edge_weight=1.0, pos_kept=None, token_lookback=3, scrubber=None, stopwords=None) ``` -------------------------------- ### Initialize NetworkX Graph and Edge Increment Function Source: https://derwen.ai/docs/ptr/explain_algo Imports the NetworkX library and defines a function to increment edge weights in a graph. If an edge does not exist, it is created with a weight of 1.0. ```python import networkx as nx def increment_edge (graph, node0, node1): ic(node0, node1) if graph.has_edge(node0, node1): graph[node0][node1]["weight"] += 1.0 else: graph.add_edge(node0, node1, weight=1.0) ``` -------------------------------- ### Generate and Upload Coverage Report Source: https://derwen.ai/docs/ptr/build Generates a test coverage report and uploads it to codecov.io. Requires a codecov.io access token stored in a '.cc_token' file. ```bash coverage report bash <(curl -s https://codecov.io/bash) -t @.cc_token ``` -------------------------------- ### Phrase Class: Initialization Source: https://derwen.ai/docs/ptr/ref Initializes a Phrase object with text, chunks, count, and rank. ```python __init__(text, chunks, count, rank) ``` -------------------------------- ### Iterate and Print Sentences Source: https://derwen.ai/docs/ptr/explain_summ Demonstrates iterating through a document's sentences and printing each one. This is a fundamental step in text processing. ```python for sent in doc.sents: ic(sent) ic| sent: Compatibility of systems of linear constraints over the set of natural numbers. ic| sent: Criteria of compatibility of a system of linear Diophantine equations, strict inequations, and nonstrict inequations are considered. ic| sent: Upper bounds for components of a minimal set of solutions and algorithms of construction of minimal generating sets of solutions for all types of systems are given. ic| sent: These criteria and the corresponding algorithms for constructing a minimal supporting set of solutions can be used in solving all the considered types systems and systems of mixed types. ``` -------------------------------- ### List generated GraphViz DOT file Source: https://derwen.ai/docs/ptr/sample Lists the details of the generated GraphViz DOT file. This command is typically run in a shell environment. ```bash !ls -lth lemma_graph.dot ```