BERTopic

https://maartengr.github.io/BERTopic/
Admin
BERTopic is a topic modeling technique and Python library that leverages transformers and c-TF-IDF...
Tokens:407,657
Snippets:1,827
Trust Score:9.7
Update:1 day ago
Show doc for...
Tokens:
Raw
### Install tiktoken and openai packages

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Install the required packages for the example. Run this command in your terminal.

```bash
pip install tiktoken openai
```

--------------------------------

### LiteLLM Usage Example

Source: https://maartengr.github.io/BERTopic/api/representations.html

Shows how to set up and use the LiteLLM representation model with BERTopic, including environment setup and custom prompt usage.

```APIDOC
## LiteLLM Usage Example

### Description
Demonstrates how to configure and use the LiteLLM representation model within BERTopic.

### Setup
```python
import os
from bertopic.representation import LiteLLM
from bertopic import BERTopic

# set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"

# Create your representation model
representation_model = LiteLLM(model="gpt-3.5-turbo")

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

### Custom Prompt
```python
prompt = "I have the following documents: [DOCUMENTS] \nThese documents are about the following topic: '"
representation_model = LiteLLM(model="gpt", prompt=prompt)
```
```

--------------------------------

### Install llama-cpp-python with CUDA acceleration

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Installation with hardware acceleration using CMAKE_ARGS and FORCE_CMAKE. Note that other installation options exist depending on your hardware and OS.

```bash
CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python
```

--------------------------------

### Install LangChain and OpenAI

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Install the required packages for using LangChain with BERTopic.

```bash
pip install langchain, openai
```

--------------------------------

### Install llama-cpp-python

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Basic installation of the llama-cpp-python package via pip.

```bash
pip install llama-cpp-python
```

--------------------------------

### Install langchain and openai packages

Source: https://maartengr.github.io/BERTopic/api/representations.html

Install the required packages before using the LangChain representation model.

```bash
pip install langchain
```

```bash
pip install openai
```

--------------------------------

### Rendered prompt example

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Shows how the prompt template is rendered after replacing [DOCUMENTS] and [KEYWORDS] with actual topic data. This is an example output, not a code snippet to run.

```text
"""
I have a topic that contains the following documents:
- Our videos are also made possible by your support on patreon.co.
- If you want to help us make more videos, you can do so on patreon.com or get one of our posters from our shop.
- If you want to help us make more videos, you can do so there.
- And if you want to support us in our endeavor to survive in the world of online video, and make more videos, you can do so on patreon.com.

The topic is described by the following keywords: videos video you our support want this us channel patreon make on we if facebook to patreoncom can for and more watch

Based on the above information, can you give a short label of the topic?
"""
```

--------------------------------

### Install BERTopic with pip

Source: https://maartengr.github.io/BERTopic/

Basic installation command for BERTopic using pip.

```bash
pip install bertopic
```

--------------------------------

### Install BERTopic with optional backends

Source: https://maartengr.github.io/BERTopic/

Installation commands for optional embedding backends (flair, gensim, spacy, use) and vision support.

```bash
# Choose an embedding backend
pip install bertopic[flair, gensim, spacy, use]

# Topic modeling with images
pip install bertopic[vision]
```

--------------------------------

### Lightweight Installation

Source: https://maartengr.github.io/BERTopic/changelog.html

Install BERTopic without sentence-transformers and use a CPU-based embedding pipeline.

```APIDOC
## Lightweight Installation

### Description
Install BERTopic without sentence-transformers to reduce environment size and enable CPU-only usage.

### Installation Commands
```
pip install --no-deps bertopic
pip install --upgrade numpy hdbscan umap-learn pandas scikit-learn tqdm plotly pyyaml
```

### Usage Example
```python
from sklearn.pipeline import make_pipeline
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

pipe = make_pipeline(
    TfidfVectorizer(),
    TruncatedSVD(100)
)

topic_model = BERTopic(embedding_model=pipe)
```

### Notes
- The default embedding model is `"all-MiniLM-L6-v2"`.
- This lightweight installation avoids `sentence-transformers` and `pytorch`.
- The resulting model can run on CPU without a GPU.
```

--------------------------------

### Install OpenAI package

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Install the openai package to use OpenAI's API with BERTopic.

```bash
pip install openai
```

--------------------------------

### Example: Using topics_per_class

Source: https://maartengr.github.io/BERTopic/api/bertopic.html

Example from the BERTopic API docs showing how to fit a model and generate topics per class.

```python
from bertopic import BERTopic
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
topics_per_class = topic_model.topics_per_class(docs, classes)
```

--------------------------------

### Install Cohere

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Install the Cohere Python package using pip.

```bash
pip install cohere
```

--------------------------------

### Install model2vec with distillation dependencies

Source: https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html

Install model2vec with the additional dependencies required for distillation.

```bash
pip install model2vec[distill]
```

--------------------------------

### Install litellm package

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Install the litellm package, which is required to use LiteLLM with BERTopic.

```bash
pip install litellm
```

--------------------------------

### Get text-based hierarchy tree with topic_model.get_topic_tree

Source: https://maartengr.github.io/BERTopic/getting_started/hierarchicaltopics/hierarchicaltopics.html

Use get_topic_tree to create a text-based representation of the hierarchy. The example shows the output when printing the tree, which helps identify topics that could be logically merged.

```python
>>> tree = topic_model.get_topic_tree(hierarchical_topics)
>>> print(tree)
.
└─atheists_atheism_god_moral_atheist
     ├─atheists_atheism_god_atheist_argument
     │    ├─■──atheists_atheism_god_atheist_argument ── Topic: 21
     │    └─■──br_god_exist_genetic_existence ── Topic: 124
     └─■──moral_morality_objective_immoral_morals ── Topic: 29
```

--------------------------------

### CohereBackend basic usage

Source: https://maartengr.github.io/BERTopic/api/backends.html

Shows how to instantiate CohereBackend with a Cohere client. Requires an API key.

```python
import cohere
from bertopic.backend import CohereBackend

client = cohere.Client("APIKEY")
cohere_model = CohereBackend(client)
```

--------------------------------

### Use a Runnable with LangChain Expression Language

Source: https://maartengr.github.io/BERTopic/api/representations.html

Create a privacy-preserving chain using LangChain Expression Language and Microsoft Presidio. The example shows imports and setup, but the prompt and llm are not fully defined.

```python
from bertopic.representation import LangChain
from langchain.chains.question_answering import load_qa_chain
from langchain.chat_models import ChatAnthropic
from langchain.schema.document import Document
from langchain.schema.runnable import RunnablePassthrough
from langchain_experimental.data_anonymizer.presidio import PresidioReversibleAnonymizer

prompt = ...
llm = ...

# We will construct a special privacy-preserving chain using Microsoft Presidio

pii_handler = PresidioReversibleAnonymizer(analyzed_fields=["PERSON"])
```

--------------------------------

### Speed up BERTopic with cuML GPU acceleration

Source: https://maartengr.github.io/BERTopic/faq.html

Use cuML's GPU-accelerated UMAP and HDBSCAN to speed up BERTopic. Requires a GPU and cuML installed. The example creates UMAP and HDBSCAN instances and passes them to BERTopic.

```python
from bertopic import BERTopic
from cuml.cluster import HDBSCAN
from cuml.manifold import UMAP

# Create instances of GPU-accelerated UMAP and HDBSCAN
umap_model = UMAP(n_components=5, n_neighbors=15, min_dist=0.0)
hdbscan_model = HDBSCAN(min_samples=10, gen_min_span_tree=True, prediction_data=True)

# Pass the above models to be used in BERTopic
topic_model = BERTopic(umap_model=umap_model, hdbscan_model=hdbscan_model)
```

--------------------------------

### Install cuML and BERTopic for CUDA 13

Source: https://maartengr.github.io/BERTopic/getting_started/dim_reduction/dim_reduction.html

Installation commands for cuML with CUDA 13. Install cuML first, then BERTopic, to avoid pip resolver issues.

```bash
!pip install cuml-cu13
!pip install bertopic
```

--------------------------------

### Install cuML and BERTopic for CUDA 12

Source: https://maartengr.github.io/BERTopic/getting_started/dim_reduction/dim_reduction.html

Installation commands for cuML with CUDA 12. Install cuML first, then BERTopic, to avoid pip resolver issues.

```bash
!pip install cuml-cu12
!pip install bertopic
```

--------------------------------

### OpenAIBackend usage example

Source: https://maartengr.github.io/BERTopic/api/backends.html

Shows how to instantiate an OpenAIBackend with an OpenAI client and the default embedding model 'text-embedding-ada-002'.

```python
import openai
from bertopic.backend import OpenAIBackend

client = openai.OpenAI(api_key="sk-...")
openai_embedder = OpenAIBackend(client, "text-embedding-ada-002")
```

--------------------------------

### Install model2vec

Source: https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html

Install the model2vec package via pip.

```bash
pip install model2vec
```

--------------------------------

### LiteLLM custom prompt example

Source: https://maartengr.github.io/BERTopic/api/representations.html

Shows how to use a custom prompt with LiteLLM by passing a prompt string to the model.

```python
prompt = "I have the following documents: [DOCUMENTS] \nThese documents are about the following topic: '"
representation_model = LiteLLM(model="gpt", prompt=prompt)
```

--------------------------------

### Lightweight installation without SentenceTransformers

Source: https://maartengr.github.io/BERTopic/changelog.html

Install BERTopic without SentenceTransformers for a lightweight package. Use the two pip commands in sequence: first install without dependencies, then upgrade the required packages.

```bash
pip install --no-deps bertopic
```

```bash
pip install --upgrade numpy hdbscan umap-learn pandas scikit-learn tqdm plotly pyyaml
```

--------------------------------

### Basic visualization with visualize_documents

Source: https://maartengr.github.io/BERTopic/api/plotting/documents.html

Minimal example to visualize documents and topics. Note that this re-calculates embeddings and reduces them to 2D.

```python
topic_model.visualize_documents(docs)
```

--------------------------------

### LangChainBackend usage example

Source: https://maartengr.github.io/BERTopic/api/backends.html

Shows how to instantiate LangChainBackend with a HuggingFaceInstructEmbeddings model from langchain_community.embeddings.

```python
from langchain_community.embeddings import HuggingFaceInstructEmbeddings
from bertopic.backend import LangChainBackend

hf_embedding = HuggingFaceInstructEmbeddings()
langchain_embedder = LangChainBackend(hf_embedding)
```

--------------------------------

### Prepare data with 20 Newsgroups

Source: https://maartengr.github.io/BERTopic/getting_started/topicsperclass/topicsperclass.html

Loads the 20 Newsgroups dataset, removes headers/footers/quotes, and creates lists of documents, targets, target names, and class labels for each document.

```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups

data = fetch_20newsgroups(subset='all',  remove=('headers', 'footers', 'quotes'))
docs = data["data"]
targets = data["target"]
target_names = data["target_names"]
classes = [data["target_names"][i] for i in data["target"]]
```

--------------------------------

### Set up BERTopic with Llama TextGeneration representation model

Source: https://maartengr.github.io/BERTopic/getting_started/representation/llm

Creates a TextGeneration representation model using the Llama generator and prompt, then initializes a BERTopic model with verbose=True.

```python
from bertopic.representation import TextGeneration
from bertopic import BERTopic

# Text generation with Llama
llama2 = TextGeneration(generator, prompt=prompt)
representation_model = {
    "Llama2": llama2,
}

# Create our BERTopic model
topic_model = BERTopic(representation_model=representation_model,  verbose=True)
```

--------------------------------

### Get frequency of all topics

Source: https://maartengr.github.io/BERTopic/api/bertopic.html

Call get_topic_freq() without arguments to get a DataFrame with the frequencies of all topics.

```python
frequency = topic_model.get_topic_freq()
```
Tokens:
Raw