### Install Toponymy from Source
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/installation.rst
Installs the latest version of Toponymy directly from its GitHub repository. This involves cloning the repository, installing dependencies, and then performing an editable installation.
```shell
git clone https://github.com/TutteInstitute/toponymy.git
cd toponymy
pip install -r requirements.txt
pip install -e .
```
--------------------------------
### Install Cloud LLM Provider Dependencies
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/installation.rst
Installs packages for integrating with cloud-based LLM services. Separate packages are available for OpenAI, Cohere, and Anthropic.
```shell
pip install openai
pip install cohere
pip install anthropic
```
--------------------------------
### Install Toponymy using conda
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/installation.rst
Installs the Toponymy library using conda, a package and environment manager. This option is useful for users who prefer conda environments.
```shell
conda install -c conda-forge toponymy
```
--------------------------------
### Install Local LLM Dependencies
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/installation.rst
Installs packages required for using local LLM models with Toponymy. 'llama-cpp-python' enables LlamaCpp models, and 'transformers' enables HuggingFace models.
```shell
pip install llama-cpp-python
pip install transformers
```
--------------------------------
### Configure OpenAI Embedder for Getting Started
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/embedding_wrappers.rst
This configuration uses the OpenAIEmbedder for getting started. It is simple, reliable, and cost-effective. It requires an OpenAI API key and specifies the model to be used.
```python
from toponymy.embedding_wrappers import OpenAIEmbedder
embedder = OpenAIEmbedder(
api_key="your-openai-api-key",
model="text-embedding-3-small"
)
```
--------------------------------
### Install Toponymy from Source
Source: https://github.com/tutteinstitute/toponymy/blob/main/README.rst
Installs the latest version of Toponymy directly from its GitHub repository. This involves cloning the repo, navigating to the directory, and running the setup.
```bash
git clone https://github.com/TutteInstitute/toponymy
cd toponymy
pip install .
```
--------------------------------
### Install Azure AI Foundry Dependency
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/installation.rst
Installs the necessary package for using models hosted on Azure's AI Foundry. This enables integration with Azure's AI services.
```shell
pip install azure-ai-inference
```
--------------------------------
### Import Toponymy Components and Initialize Models
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Imports the Toponymy class, clusterer, keyphrase builder, LLM wrappers, and sentence transformer for text embeddings. It also initializes the embedding model and reads an Azure AI API key. This setup is crucial for instantiating the Toponymy topic model.
```python
from toponymy import Toponymy, ToponymyClusterer, KeyphraseBuilder
from toponymy.llm_wrappers import AzureAINamer
from sentence_transformers import SentenceTransformer
embedding_model = SentenceTransformer("paraphrase-MiniLM-L3-v2")
azure_api_key = open("../azure_cohere_api_key.txt").read().strip()
```
--------------------------------
### Instantiate Toponymy Topic Model
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Creates an instance of the Toponymy class, configuring it with an LLM wrapper (AzureAI Cohere model), a text embedding model, a clusterer, and a keyphrase builder. It also sets object and corpus descriptions, and specifies custom exemplar delimiters for handling messy text data.
```python
topic_model = Toponymy(
llm_wrapper=AzureAINamer(
azure_api_key,
endpoint="https://azureaitimcuse5821437469.services.ai.azure.com/models",
model="Cohere-command-r-08-2024",
),
text_embedding_model=embedding_model,
clusterer=ToponymyClusterer(min_clusters=4, verbose=True),
keyphrase_builder=KeyphraseBuilder(ngram_range=(1,6), max_features=15_000, verbose=True),
object_description="newsgroup posts",
corpus_description="20-newsgroups dataset",
exemplar_delimiters=["\n","\n\n\n"],
)
```
--------------------------------
### Build Documentation Locally with Sphinx
Source: https://github.com/tutteinstitute/toponymy/blob/main/CONTRIBUTING.md
Installs documentation tool requirements and builds the documentation in HTML format using Sphinx. The output is located in the 'doc/_build' folder.
```bash
pip install -r doc/requirements.txt
sphinx-build -b html doc doc/_build
```
--------------------------------
### Install Toponymy Package
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/index.rst
This snippet shows the command to install the Toponymy package using pip. It is a prerequisite for using the library.
```bash
pip install toponymy
```
--------------------------------
### Install Packages using Pip
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_max_layers_newsgroups.ipynb
Installs the pandas and pyarrow libraries, which are required for data manipulation and efficient data handling, respectively. This is a prerequisite for running the subsequent code in the notebook.
```python
# Install required packages if needed
!pip install pandas pyarrow
```
--------------------------------
### Importing Datamapplot Libraries
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Imports the necessary datamapplot libraries for creating interactive plots. This is the initial step required before utilizing any of datamapplot's plotting functionalities.
```python
import datamapplot
import datamapplot.selection_handlers
```
--------------------------------
### Import Toponymy Components and Load API Keys
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/how_toponymy_works.ipynb
Imports necessary classes from the toponymy library and loads Azure API keys from a file. This setup is required before initializing the Toponymy model and using LLM services.
```python
from toponymy.toponymy import Toponymy, ToponymyClusterer, KeyphraseBuilder
from toponymy.cluster_layer import ClusterLayerText
from toponymy.llm_wrappers import AzureAINamer
from toponymy.embedding_wrappers import AzureAIEmbedder
azure_api_key = open("../azure_cohere_api_key.txt").read().strip()
```
--------------------------------
### Build Jupyter Widget using npm
Source: https://github.com/tutteinstitute/toponymy/blob/main/toponymy/widgets/README.md
Commands to build a Jupyter notebook widget. This process involves navigating to the widget's subdirectory, installing its dependencies using npm, and then running the build script. The output is placed in the `dist/` folder.
```bash
cd widgets/
npm install
npm run build
```
--------------------------------
### Test LLM Connectivity with OpenAI
Source: https://github.com/tutteinstitute/toponymy/blob/main/README.rst
This snippet demonstrates how to initialize the OpenAI LLM wrapper and test the connection to the OpenAI API. It requires an 'openai_key.txt' file containing the API key and the 'openai' Python package to be installed.
```python
import openai
from toponymy import Toponymy
from toponymy.llm_wrappers import OpenAI
openai_api_key = open("openai_key.txt").read().strip()llm = OpenAI(openai_api_key)llm.test_llm_connectivity()
```
--------------------------------
### Local Model Recommendations for Topic Naming (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
Provides Python examples for using local language models via Ollama, HuggingFace, and VLLM for topic naming. It includes recommendations for various model sizes and providers, suitable for different resource constraints.
```python
# Ollama models - easy installation and management
llm = OllamaNamer(model="llama3.2") # Latest Llama model via Ollama
llm = OllamaNamer(model="mistral") # Mistral model via Ollama
llm = OllamaNamer(model="qwen2.5") # Qwen model via Ollama
# 7B models via HuggingFace/VLLM - good balance of quality and resource requirements
llm = HuggingFaceNamer(model="mistralai/Mistral-7B-Instruct-v0.3")
llm = VLLMNamer(model="mistralai/Mistral-7B-Instruct-v0.3")
# 13B models - better quality, higher resource requirements
llm = HuggingFaceNamer(model="mistralai/Mixtral-8x7B-Instruct-v0.1")
# Smaller models - for resource-constrained environments
llm = OllamaNamer(model="llama3.2:1b") # 1B parameter model via Ollama
```
--------------------------------
### Install OpenAI Library for Embedding Wrapper
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/embedding_wrappers.rst
Installs the necessary 'openai' library to enable the OpenAI embedding wrapper within Toponymy. This is a prerequisite for using OpenAI's embedding models.
```bash
pip install openai
```
--------------------------------
### Format Code with Black
Source: https://github.com/tutteinstitute/toponymy/blob/main/CONTRIBUTING.md
Installs and runs the Black code formatter to ensure consistent code style across the project. This command should be executed in the root of the project directory.
```bash
pip install black
black .
```
--------------------------------
### Displaying Progress Bars for Tasks (Shell)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Shows progress bars for various computational tasks within the Toponymy project, such as selecting central exemplars, building topic names, generating keyphrases, and generating prompts. These visualizations help in understanding the real-time progress of complex operations and provide an estimate of the remaining time.
```shell
Selecting central exemplars: 0%| | 0/429 [00:00, "topic_specificity":}
where NAME is the topic name you generate and SCORE is a float value between 0.0 and 1.0,
representing how specific and well-defined the topic name is given the input information.
A score of 1.0 means a perfectly descriptive and specific name, while 0.0 would be a completely generic or unrelated name.
You should primarily make use of the major and minor subtopics of this group to generate a name,
and ensure the topic name reflects the core essence of *all* major subtopics.
Ensure your entire response is only the JSON object, with no other text before or after it.
```
--------------------------------
### Initialize and Use MistralEmbedder
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/embedding_wrappers.rst
Provides an example of initializing the MistralEmbedder with an API key and specifying the embedding model. Mistral's API offers competitive performance and pricing.
```python
from toponymy.embedding_wrappers import MistralEmbedder
# Initialize with Mistral API
embedder = MistralEmbedder(
api_key="your-mistral-api-key",
model="mistral-embed" # Mistral's embedding model
)
# Generate embeddings
embeddings = embedder.encode(
texts=["natural language processing", "text mining", "information retrieval"],
show_progress_bar=True
)
```
--------------------------------
### Initialize Embedding and LLM Models - Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
Initializes the SentenceTransformer embedding model and the OpenAI LLM wrapper for topic naming. It demonstrates how to set up the OpenAI API key, either directly, via environment variables, or from a file, and configures the LLM wrapper with a specific model.
```python
# Initialize embedding model
print("Loading embedding model...")
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
# Initialize OpenAI LLM wrapper
import os
# IMPORTANT: Replace with your actual OpenAI API key
# You can get one at: https://platform.openai.com/api-keys
#openai_api_key = "sk-YOUR-API-KEY-HERE"
# Alternative: Load from environment variable (recommended for security)
#openai_api_key = os.getenv("OPENAI_API_KEY")
# Alternative: Load from a file
# with open("openai_key.txt", "r") as f:
# openai_api_key = f.read().strip()
# Import OpenAI from the correct module
import toponymy.llm_wrappers
OpenAI = toponymy.llm_wrappers.OpenAINamer
# Create OpenAI wrapper
# Default model is gpt-4o-mini which is cost-effective for topic naming
llm = OpenAINamer(api_key=openai_api_key, model="gpt-4o-mini")
print("OpenAI model initialized!")
print(f"Using model: {llm.model}")
print("Ready to generate topic names using OpenAI API")
```
--------------------------------
### Initialize AzureAINamer with Azure AI Credentials
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
This example shows how to initialize the AzureAINamer wrapper for models deployed on Azure AI. It requires an Azure API key, the service endpoint, and the deployed model name, along with custom instructions. Useful for integrating Azure-hosted models.
```python
from toponymy.llm_wrappers import AzureAINamer
# Initialize with Azure AI
llm = AzureAINamer(
api_key="your-azure-api-key",
endpoint="https://your-endpoint.inference.ai.azure.com",
model="your-deployed-model-name",
llm_specific_instructions="Generate professional topic names"
)
```
--------------------------------
### Initialize AnthropicNamer with Anthropic API Key
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
This example demonstrates initializing the AnthropicNamer wrapper for Claude models. It requires an Anthropic API key and allows specifying the model and custom instructions for generating names. Claude models are noted for following complex instructions well.
```python
from toponymy.llm_wrappers import AnthropicNamer
# Initialize with Anthropic API
llm = AnthropicNamer(
api_key="your-anthropic-api-key", # Or set ANTHROPIC_API_KEY env var
model="claude-3-haiku-20240307", # Fast and cost-effective
llm_specific_instructions="Generate coherent, descriptive names"
)
```
--------------------------------
### Initialize and Fit Toponymy Model in Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
This snippet demonstrates how to initialize the Toponymy model with specified language models, clusterers, and descriptions. It then proceeds to fit the model to a subset of text data, vectors, and a map, printing status messages and warnings about potential API costs.
```python
from toponymy import Toponymy
# Assuming llm, embedding_model, clusterer, text_subset, vectors_subset, map_subset are defined elsewhere
topic_model = Toponymy(
llm_wrapper=llm,
text_embedding_model=embedding_model,
clusterer=clusterer,
object_description="newsgroup posts",
corpus_description="20-newsgroups dataset",
exemplar_delimiters=["\n", "\n\n\n"],
)
print("Fitting Toponymy model...")
print("This will make API calls to OpenAI - costs will apply!")
topic_model.fit(text_subset, vectors_subset, map_subset)
print("Model fitted!")
```
--------------------------------
### Load and Prepare 20-Newsgroups Dataset - Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
Loads the 20-newsgroups dataset from a Parquet file using pandas. It extracts document text, precomputed embeddings (vectors), and mapping information. This snippet prepares the data for use with the Toponymy model.
```python
# Load the 20-newsgroups dataset with precomputed embeddings
print("Loading 20-newsgroups dataset...")
newsgroups_df = pd.read_parquet("hf://datasets/lmcinnes/20newsgroups_embedded/data/train-00000-of-00001.parquet")
# Extract text, vectors, and map
text = newsgroups_df["post"].str.strip().values
document_vectors = np.stack(newsgroups_df["embedding"].values)
document_map = np.stack(newsgroups_df["map"].values)
print(f"Loaded {len(text)} documents")
print(f"Document vectors shape: {document_vectors.shape}")
print(f"Document map shape: {document_map.shape}")
```
--------------------------------
### Generate Topic Modeling Prompts (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/how_toponymy_works.ipynb
This Python code snippet demonstrates the generation of prompts for topic modeling. It utilizes a TopicModel object to extract exemplar texts, keyphrases, subtopics, and finally, prompts for a specified layer. The output includes example system and user prompts.
```python
topic_model.cluster_layers_[2].make_exemplar_texts(arxiv_ct_df["title"], embedding_vectors);
topic_model.cluster_layers_[2].make_keyphrases(
topic_model.keyphrase_list_,
topic_model.object_x_keyphrase_matrix_,
keyphrase_vectors=keyphrase_vectors,
embedding_model=embedding_model
);
topic_model.cluster_layers_[2].make_subtopics(
topic_model.topic_names_[0],
topic_model.cluster_layers_[0].cluster_labels,
topic_model.cluster_layers_[0].topic_name_embeddings,
topic_model.embedding_model,
);
topic_model.cluster_layers_[2].make_prompts(
detail_level=0.66,
all_topic_names=topic_model.topic_names_,
object_description=topic_model.object_description,
corpus_description=topic_model.corpus_description,
cluster_tree=topic_model.cluster_tree_,
);
print("### System prompt:\n", topic_model.cluster_layers_[2].prompts[0]["system"])
print("\n\n### User prompt:\n", topic_model.cluster_layers_[2].prompts[0]["user"])
```
--------------------------------
### Example Disambiguation Prompts for Category Theory Topics
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/how_toponymy_works.ipynb
This section presents the output of the Python code, showing the detailed system and user prompts designed for generating specific topic names in category theory. The system prompt outlines the role and task, while the user prompt provides context and expected output format.
```text
### System prompt:
You are an expert in Algebraic Structures and Their Extensions and Abstract Algebraic Structures and Their Properties. You have been asked to provide more specific and distinguishing names for various groups of
paper titles from category theory related papers that have been assigned overly similar auto-generated topic names.
Your task is to generate a new domain expert level (8 to 15 word) name for each topic group presented.
Your should make use of the relative relationships between these topics, their keywords, subtopic information, and sample paper titles to generate new, distinct topic names.
The new names must be in the same order as the original topics are presented.
There should be no duplicate topic names in your final list of new names.
The response must be formatted as a single JSON object in the format:
{"new_topic_name_mapping": {"1. OLD_NAME1": "NEW_NAME1", "2. OLD_NAME2": "NEW_NAME2", ... }, "topic_specificities": [NEW_TOPIC_SCORE1, NEW_TOPIC_SCORE2, ...]}
where SCORE is a float value between 0.0 and 1.0 representing the quality and specificity of the new name.
Ensure your entire response is only the JSON object, with no other text before or after it.
### User prompt:
('Quasi-Coherent Sheaves and Derived Categories', 1),
('Cospan and Double Category Theory', 1),
('Frobenius-Schur Indicators in Category Theory', 1)]
```
--------------------------------
### Process Topics Concurrently with AsyncAnthropicNamer
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
This example shows how to use an asynchronous wrapper, AsyncAnthropicNamer, to process multiple prompts concurrently. It highlights setting a maximum concurrency level and includes cleanup of resources. This pattern is beneficial for improving throughput with large datasets.
```python
import asyncio
from toponymy.llm_wrappers import AsyncAnthropicNamer
async def process_topics():
llm = AsyncAnthropicNamer(
api_key="your-api-key",
max_concurrent_requests=5 # Control concurrency
)
# Process multiple prompts concurrently
prompts = [prompt1, prompt2, prompt3, ...]
results = await llm.generate_topic_names(prompts)
await llm.close() # Clean up resources
return results
# Run the async function
```
--------------------------------
### Generate Exemplars, Keyphrases, and Subtopics in Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/how_toponymy_works.ipynb
This Python code snippet demonstrates the process of generating exemplars, keyphrases, and subtopics for a given layer in a topic model. It utilizes pre-defined data and embedding models to analyze and categorize content. The output includes progress indicators and example subtopic names.
```python
topic_model.cluster_layers_[1].make_exemplar_texts(arxiv_ct_df["title"], embedding_vectors);
topic_model.cluster_layers_[1].make_keyphrases(
topic_model.keyphrase_list_,
topic_model.object_x_keyphrase_matrix_,
keyphrase_vectors=keyphrase_vectors,
embedding_model=embedding_model
);
topic_model.cluster_layers_[1].make_subtopics(
topic_model.topic_names_[0],
topic_model.cluster_layers_[0].cluster_labels,
topic_model.cluster_layers_[0].topic_name_embeddings,
topic_model.embedding_model,
);
topic_model.cluster_layers_[1].subtopics[0]
```
--------------------------------
### Getting Unique Newsgroup Names (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Extracts a unique list of newsgroup names from the newsgroups DataFrame. This is useful for understanding the sources of the text data being analyzed and for correlating topics with their origins.
```python
newsgroups_df.newsgroup.unique().tolist()
```
--------------------------------
### Load ArXiv Machine Learning Dataset
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/exemplar_texts.ipynb
Loads a portion of the ArXiv machine learning dataset using pandas. This dataset contains titles and abstracts of machine learning papers. The data is read from a parquet file stored on Hugging Face datasets.
```python
arxiv_ml_df = pd.read_parquet("hf://datasets/lmcinnes/arxiv_ml/data/train-00000-of-00008-f3c9b137f969d545.parquet")
```
--------------------------------
### Configure SentenceTransformer for Budget-Conscious Projects
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/embedding_wrappers.rst
This configuration uses SentenceTransformer for budget-conscious projects. It offers free usage after initial setup and good performance. It requires the sentence-transformers library to be installed.
```python
from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer("all-MiniLM-L6-v2")
```
--------------------------------
### Display First Cluster Details and Documents (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
This Python snippet displays the topic name, keyphrases, and a sample of documents for the first cluster in the 'audit_with_docs' DataFrame. It iterates through the documents of the first cluster and prints their content, with a limit to the number of documents shown. Dependencies include the pandas library for DataFrame manipulation.
```python
first_cluster = audit_with_docs.iloc[0]
print(f"Cluster 0: {first_cluster['llm_topic_name']}")
print(f"Keyphrases: {first_cluster['top_5_keyphrases']}")
print(f"\nAll {first_cluster['num_documents']} documents in this cluster:")
if 'document_texts' in first_cluster:
for i, (idx, doc) in enumerate(zip(first_cluster['document_indices'], first_cluster['document_texts'])):
print(f"\n[Document {idx}]:")
print(doc[:200] + "..." if len(doc) > 200 else doc)
if i >= 2: # Show only first 3 documents
print(f"\n... and {len(first_cluster['document_texts']) - 3} more documents")
break
```
--------------------------------
### OpenAI Model Recommendations for Topic Naming (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
Demonstrates how to instantiate OpenAI's topic naming models in Python. It highlights recommended cost-effective options and notes less suitable, more expensive models for this specific task.
```python
llm = OpenAINamer(model="gpt-4o-mini") # ~$0.15/1M input tokens
llm = OpenAINamer(model="gpt-4o") # ~$2.50/1M input tokens
llm = OpenAINamer(model="o1-preview") # ~$15/1M input tokens
llm = OpenAINamer(model="gpt-4") # ~$30/1M input tokens
```
--------------------------------
### Toponymy Model Object Representation (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Displays the string representation of the Toponymy model object after fitting. This typically shows the object's type and memory address, indicating that the model has been successfully instantiated and is ready for further analysis.
```python
```
--------------------------------
### Accessing Topic Tree (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Accesses the topic tree attribute of the topic model. This attribute represents the hierarchical structure of topics, allowing for exploration of how broader topics decompose into more specific ones. The output is an instance of the TopicTree class.
```python
topic_model.topic_tree_
```
--------------------------------
### Display Basic Model Information - Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
This Python code snippet displays fundamental information about the fitted topic model, specifically the number of layers and the cluster count within each layer. It iterates through the model's cluster layers to extract and print this data. No external dependencies beyond numpy are explicitly shown, but the context implies a topic modeling library is in use.
```python
# Show basic model information
print(f"Number of layers: {len(topic_model.cluster_layers_)}")
for i, layer in enumerate(topic_model.cluster_layers_):
n_clusters = len(np.unique(layer.cluster_labels)) - 1 # Exclude -1
print(f"Layer {i}: {n_clusters} clusters")
```
--------------------------------
### Initialize GoogleGeminiNamer with Google API Key
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
This snippet demonstrates initializing the GoogleGeminiNamer wrapper for Google's Gemini models. It requires a Google API key and allows specifying the model and custom instructions. Gemini models provide cost-efficient performance.
```python
from toponymy.llm_wrappers import GoogleGeminiNamer
# Initialize with Google Gemini API
llm = GoogleGeminiNamer(
api_key="your-google-api-key", # Or set GOOGLE_API_KEY env var
model="gemini-1.5-flash", # Fast and cost-effective
llm_specific_instructions="Generate concise, relevant topic names"
)
```
--------------------------------
### Accessing Topic Names (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Retrieves the names of topics from a topic model at a specified resolution layer. This is useful for understanding the thematic content captured by the model at different levels of granularity. The output provides a list of topic names.
```python
topic_model.topic_names_[-2]
```
--------------------------------
### Analyze Keyphrase to Topic Name Mapping in Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
This Python code snippet analyzes the relationship between keyphrases and topic names within a topic model. It generates a DataFrame showing this mapping, displays the first 20 entries, and calculates summary statistics on keyphrase usage in topic names. It relies on a function `create_keyphrase_analysis_df` and assumes the existence of `topic_model` and `display` (likely from IPython).
```python
# Analyze how keyphrases relate to topic names
keyphrase_df = create_keyphrase_analysis_df(topic_model, layer_index=0)
# Show some examples
print("\nKeyphrase to Topic Name Mapping (first 20):")
display(keyphrase_df.head(20))
# Summary statistics
keyphrase_usage = keyphrase_df['keyphrase_in_topic'].value_counts()
print(f"\nKeyphrase usage in topic names:")
print(f"Keyphrases appearing in topic names: {keyphrase_usage.get(True, 0)}")
print(f"Keyphrases NOT in topic names: {keyphrase_usage.get(False, 0)}")
print(f"Percentage of keyphrases used: {keyphrase_usage.get(True, 0) / len(keyphrase_df) * 100:.1f}%")
```
--------------------------------
### Configure Topic Naming Environment
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/arxiv_topic_labelling.ipynb
Sets up the Python environment by appending a topic naming directory to the system path and importing the Toponymy class. This is crucial for accessing custom modules required for topic analysis.
```python
import sys
topic_path=Path("/work/home/jchealy/STAMP2024/TopicNaming/")
sys.path.append(str(topic_path))
from topic_naming import Toponymy
```
--------------------------------
### Display Audit DataFrame (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
This Python snippet simply prints the 'audit_with_docs' DataFrame, which contains statistics and keyphrases for various clusters. It's useful for getting an overview of the clustering results. The output is a tabular representation of the data.
```python
print(audit_with_docs)
```
--------------------------------
### Accessing Top Topic Names (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Retrieves the first ten topic names from the most fine-grained resolution layer (layer 0) of the topic model. This allows for a detailed examination of the specific themes identified by the model, illustrating its ability to capture niche subjects.
```python
topic_model.topic_names_[0][:10]
```
--------------------------------
### Configure and Initialize Toponymy Model
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/how_toponymy_works.ipynb
Initializes the Toponymy model with specified configurations. This includes setting the LLM wrapper, text embedding model, cluster layer class, clusterer, keyphrase builder, and descriptive attributes for the object and corpus. It demonstrates a step-by-step approach to fitting the model.
```python
topic_model = Toponymy(
llm_wrapper=AzureAINamer(
azure_api_key,
endpoint="https://azureaitimcuse5821437469.services.ai.azure.com/models",
model="Cohere-command-r-08-2024",
),
text_embedding_model=embedding_model,
layer_class=ClusterLayerText,
clusterer=ToponymyClusterer(min_clusters=4, verbose=True),
keyphrase_builder=KeyphraseBuilder(ngram_range=(1,6), max_features=15_000, verbose=True),
object_description="paper titles",
corpus_description="category theory related papers",
)
```
--------------------------------
### Python: Get Document Text List Length
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
This Python code snippet calculates and returns the number of items in a specific document text list within the 'audit_with_docs' data structure. It's a simple operation to determine the size of a dataset.
```python
len(audit_with_docs['document_texts'][0])
```
--------------------------------
### Analyze LLM Prompts in Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
This Python code snippet is designed to analyze the prompts sent to a Large Language Model (LLM) within the context of a topic model. It utilizes a function `create_prompt_analysis_df` to generate a DataFrame containing prompt analysis results and then displays the first 10 entries of this DataFrame. This is useful for understanding how the topic model is interacting with the LLM.
```python
# Analyze prompts sent to LLM
prompt_df = create_prompt_analysis_df(topic_model)
print("\nPrompt Analysis:")
display(prompt_df.head(10))
```
--------------------------------
### Trace Keyphrases to Documents - Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
Traces keyphrases from a specific cluster back to their source documents within that cluster. This function helps understand which documents contribute to particular keyphrases. It uses the topic model and a helper function to get cluster documents.
```python
# Trace keyphrases back to their source documents
layer = topic_model.cluster_layers_[0]
# Pick a cluster to analyze
cluster_id = 0
print(f"Analyzing Cluster {cluster_id}: {layer.topic_names[cluster_id]}")
# Get keyphrases and documents for this cluster
keyphrases = layer.keyphrases[cluster_id][:5] # Top 5 keyphrases
cluster_docs = get_cluster_documents(topic_model, 0, cluster_id, original_texts)
print(f"\nTop 5 keyphrases: {', '.join(keyphrases)}")
print(f"Checking these keyphrases in {cluster_docs['total_count']} cluster documents:")
```
--------------------------------
### Accessing Top-Level Topic Names (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Retrieves the names of the top-level topics discovered by the model. The `topic_names_` attribute is a list of lists, where each inner list represents topic names at a specific granularity layer. Accessing the last element (`[-1]`) provides the most generalized topic names.
```python
topic_model.topic_names_[-1]
```
--------------------------------
### Exemplars from First Cluster (Diverse)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/exemplar_texts.ipynb
Displays a selection of exemplar titles obtained from the first cluster using the `diverse_exemplars` method. These titles relate to polynomial systems solving and linear algebra, indicating a focus on mathematical and computational topics.
```plaintext
['Polynomial Systems Solving by Fast Linear Algebra',
'Algorithms for Computing Triangular Decompositions of Polynomial Systems',
'Constructive $D$-module Theory with \textsc{Singular}',
'On the Complexity of the Multivariate Resultant',
'Real Polynomial Root-finding by Means of Matrix and Polynomial Iterations',
'Fast Computation of the Roots of Polynomials Over the Ring of Power Series',
'Critical Points and Gr\"obner Bases: the Unmixed Case',
'On the asymptotic and practical complexity of solving bivariate systems over the reals']
```
--------------------------------
### Creating an Interactive Document Data Map
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Generates an interactive plot visualizing document-topic relationships. This function takes 2D coordinates, topic name vectors, and several optional parameters for customization, including titles, hover text, marker size, colormaps, search, and selection handlers.
```python
plot = datamapplot.create_interactive_plot(
clusterable_vectors,
*topic_model.topic_name_vectors_,
title="20-Newsgroups",
sub_title="A data map of 20-newsgroups using all-mpnet-basev2, Toponymy, Cohere and UMAP",
hover_text=newsgroups_df["post"].values,
font_family="Cormorant SC",
marker_size_array=np.asarray([np.log(len(x)) for x in newsgroups_df["post"].values]),
colormaps={"newsgroup": pd.Series(newsgroups_df["newsgroup"].values)},
cluster_layer_colormaps=True,
enable_search=True,
selection_handler=datamapplot.selection_handlers.WordCloud(height=300),
)
plot
```
--------------------------------
### Prepare Embedding and Clusterable Vectors (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Extracts and prepares embedding vectors and clusterable vectors from the newsgroups DataFrame. It includes a conditional block to generate these vectors from raw text using Sentence Transformers and UMAP if needed, though the primary execution path uses pre-computed vectors.
```python
if False:
from sentence_transformers import SentenceTransformer
from umap import UMAP
embedding_model = SentenceTransformer("all-mpnet-base-v2")
embedding_vectors = embedding_model.encode(newsgroups_df["post"], show_progress_bar=True)
clusterable_vectors = UMAP(metric="cosine").fit_transform(embedding_vectors)
else:
embedding_vectors = np.stack(newsgroups_df["embedding"].values)
clusterable_vectors = np.stack(newsgroups_df["map"].values)
```
--------------------------------
### Initialize LlamaCppNamer for Local LLM Inference
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
This Python code initializes the LlamaCppNamer wrapper for using local GGUF format models. It requires the model file path and allows configuration of context window size and GPU layer offloading. Note that LlamaCpp does not support system prompts.
```python
from toponymy.llm_wrappers import LlamaCppNamer
# Initialize with a local model file
llm = LlamaCppNamer(
model_path="/path/to/your/model.gguf",
llm_specific_instructions="Be concise and descriptive",
n_ctx=4096, # Context window size
n_gpu_layers=35 # Number of layers to offload to GPU
)
```
--------------------------------
### Select LLM Wrapper for Topic Naming Workflows
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
This section demonstrates selecting appropriate LLM wrappers from the toponymy library for different scales of topic naming tasks. It covers initialization for exploration, production, large-scale batch processing, and privacy-sensitive local processing, with specific model recommendations for each.
```python
# For exploration and small datasets
from toponymy.llm_wrappers import AnthropicNamer
llm = AnthropicNamer(api_key="...", model="claude-3-haiku-20240307")
```
```python
# For production with moderate scale
from toponymy.llm_wrappers import AsyncOpenAINamer
llm = AsyncOpenAINamer(api_key="...", model="gpt-4o-mini", max_concurrent_requests=5)
```
```python
# For large-scale batch processing
from toponymy.llm_wrappers import BatchAnthropicNamer
llm = BatchAnthropicNamer(api_key="...", model="claude-3-haiku-20240307")
```
```python
# For privacy-sensitive local processing
from toponymy.llm_wrappers import OllamaNamer
llm = OllamaNamer(model="llama3.2")
```
```python
# For cost-conscious processing with good performance
from toponymy.llm_wrappers import GoogleGeminiNamer
llm = GoogleGeminiNamer(api_key="...", model="gemini-1.5-flash")
```
--------------------------------
### Import Matplotlib for Plotting
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/exemplar_texts.ipynb
Imports the matplotlib.pyplot module, which is essential for creating visualizations. This is a standard first step for any plotting task in Python.
```python
import matplotlib.pyplot as plt
```
--------------------------------
### Get Cluster Documents - Python
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/test_audit_functionality.ipynb
Retrieves all documents associated with a specific cluster ID from a topic model. This helper function allows detailed examination of documents within a cluster, providing indices, texts, and total counts. It requires the topic model, layer index, cluster ID, and original texts.
```python
# Import the new helper function
from toponymy.audit import get_cluster_documents
# Get all documents for a specific cluster
cluster_id = 2 # Choose a cluster to examine
cluster_docs = get_cluster_documents(
topic_model,
layer_index=0,
cluster_id=cluster_id,
original_texts=original_texts
)
print(f"Cluster {cluster_id}: {topic_model.cluster_layers_[0].topic_names[cluster_id]}")
print(f"Total documents: {cluster_docs['total_count']}")
print(f"\nDocument indices: {cluster_docs['indices']}")
print(f"\nFirst 3 documents:")
for idx, text in zip(cluster_docs['indices'][:3], cluster_docs['texts'][:3]):
print(f"\n[Document {idx}]:")
print(text[:200] + "..." if len(text) > 200 else text)
```
--------------------------------
### Displaying Progress and Status Messages (Shell)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Represents progress and status updates during the execution of the Toponymy project's processes. These messages typically indicate the number of clusters found at different layers, the progress of keyphrase matrix building, chunking, and count dictionary combination. They are useful for monitoring the execution flow and identifying potential bottlenecks.
```shell
Layer 0 found 429 clusters
Layer 1 found 134 clusters
Layer 2 found 41 clusters
Layer 3 found 14 clusters
Layer 4 found 5 clusters
Building keyphrase matrix ...
Chunking into 1 chunks of size 20000 for keyphrase identification.
Combining count dictionaries ...
Found 15000 keyphrases.
Chunking into 1 chunks of size 20000 for keyphrase count construction.
Combining count matrix chunks ...
CPU times: user 36.5 s, sys: 2.87 s, total: 39.4 s
Wall time: 19min 4s
```
--------------------------------
### Initialize Google Gemini LLM for Topic Naming
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/llm_wrappers.rst
This code snippet initializes a Google Gemini LLM wrapper for topic naming. It specifies the model to be used, such as 'gemini-1.5-pro' or 'gemini-1.5-flash'. The cost associated with input tokens is also noted.
```python
llm = GoogleGeminiNamer(model="gemini-1.5-pro") # ~$1.25/1M input tokens
```
```python
llm = GoogleGeminiNamer(api_key="...", model="gemini-1.5-flash")
```
--------------------------------
### Fitting Topic Model with Text Data and Embeddings (Python)
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/basic_usage.ipynb
Fits a topic model using provided text data, pre-computed embedding vectors, and clusterable vectors. This function is central to the topic modeling process in the Toponymy project. It requires the text data to be preprocessed (e.g., stripped of whitespace) and the embedding and clusterable vectors to be correctly generated.
```python
topic_model.fit(
newsgroups_df["post"].str.strip().values,
embedding_vectors=embedding_vectors,
clusterable_vectors=clusterable_vectors
)
```
--------------------------------
### Create Cluster Layers with ToponymyClusterer
Source: https://github.com/tutteinstitute/toponymy/blob/main/doc/exemplar_texts.ipynb
Initializes and fits the ToponymyClusterer to the document and clusterable vectors. This process generates hierarchical cluster layers at different resolutions and outputs the cluster structure.
```python
layers, tree = ToponymyClusterer(verbose=True).fit_predict(clusterable_vectors=clusterable_vectors, embedding_vectors=document_vectors)
```