### Quick Start Example

Source: https://bitermplus.readthedocs.io/en/latest/sklearn_api.html

Demonstrates how to quickly create, fit, and use the BTMClassifier model with sample documents.

```python
import bitermplus as btm

# Sample documents
texts = [
    "machine learning algorithms are powerful",
    "deep learning neural networks process data",
    "natural language processing understands text",
    "artificial intelligence transforms industries"
]

# Create and fit model (one step!)
model = btm.BTMClassifier(n_topics=2, random_state=42)
model.fit(texts)

# Get topic distributions
doc_topics = model.transform(texts)
print(f"Document-topic matrix shape: {doc_topics.shape}")

# Interpret topics
topic_words = model.get_topic_words(n_words=5)
for topic_id, words in topic_words.items():
    print(f"Topic {topic_id}: {', '.join(words)}")

```

--------------------------------

### Install bitermplus from PyPI

Source: https://bitermplus.readthedocs.io/en/latest/install.html

This command installs the stable version of bitermplus from the Python Package Index.

```bash
pip install bitermplus

```

--------------------------------

### Install bitermplus from PyPI

Source: https://bitermplus.readthedocs.io/en/latest/_sources/install.rst.txt

This command installs the stable version of the bitermplus library from the Python Package Index (PyPI).

```bash
pip install bitermplus
```

--------------------------------

### Install latest development version from GitHub

Source: https://bitermplus.readthedocs.io/en/latest/_sources/install.rst.txt

This command installs the latest development version of bitermplus directly from its GitHub repository.

```bash
pip install git+https://github.com/maximtrp/bitermplus.git
```

--------------------------------

### Install latest development version from GitHub

Source: https://bitermplus.readthedocs.io/en/latest/install.html

This command installs the latest development version of bitermplus directly from its GitHub repository.

```bash
pip install git+https://github.com/maximtrp/bitermplus.git

```

--------------------------------

### Example for get_top_topic_docs

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Example demonstrating how to select top topic documents from a fitted BTM model.

```python
>>> top_docs = btm.get_top_topic_docs(
...     texts,
...     p_zd,
...     docs_num=100,
...     topics_idx=[1,2,3,4])

```

--------------------------------

### Model Fitting

Source: https://bitermplus.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Demonstrates basic model fitting, including importing data, vectorizing documents, extracting biterms, initializing, and running the BTM model. It also shows an example of stop word removal during vectorization.

```python
import bitermplus as btm
import numpy as np
import pandas as pd

# Importing data
df = pd.read_csv(
    'dataset/SearchSnippets.txt.gz', header=None, names=['texts'])
texts = df['texts'].str.strip().tolist()

# Vectorize documents and extract biterms
# Uses sklearn's CountVectorizer internally - accepts its parameters
# Example: stop word removal
stop_words = ["word1", "word2", "word3"]
X, vocabulary, vocab_dict = btm.get_words_freqs(texts, stop_words=stop_words)
docs_vec = btm.get_vectorized_docs(texts, vocabulary)
biterms = btm.get_biterms(docs_vec)

# Initializing and running model
model = btm.BTM(
    X, vocabulary, seed=12321, T=8, M=20, alpha=50/8, beta=0.01)
model.fit(biterms, iterations=20)
```

--------------------------------

### Model Initialization and Fitting

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.html

Example of initializing a BTM model and fitting it with biterms.

```python
>>> import bitermplus as btm
>>> # Assume biterms is prepared
>>> model = btm.BTM(X, vocabulary, T=5)
>>> model.fit(biterms, iterations=200, verbose=True)
```

--------------------------------

### Example for get_docs_top_topic

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Example showing how to find the most probable topic for each document.

```python
>>> import bitermplus as btm
>>> # Read documents from file
>>> # texts = ...
>>> # Build and train a model
>>> # model = ...
>>> # model.fit(...)
>>> btm.get_docs_top_topic(texts, model.matrix_docs_topics_)

```

--------------------------------

### Basic Usage Example

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

Demonstrates how to initialize, fit, and transform documents using the BTMClassifier.

```python
import bitermplus as btm

texts = [
    "machine learning algorithms are powerful",
    "deep learning neural networks process data",
    "natural language processing understands text"
]

model = btm.BTMClassifier(n_topics=2, random_state=42)
model.fit(texts)

doc_topics = model.transform(texts)
print(f"Shape: {doc_topics.shape}")
```

--------------------------------

### Getting Topic Words Example

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

Shows how to retrieve the top words associated with each topic from a fitted model.

```python
topic_words = model.get_topic_words(n_words=5)
for topic_id, words in topic_words.items():
    print(f"Topic {topic_id}: {', '.join(words)}")
```

--------------------------------

### Entropy Calculation Example

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.metrics.html

Example demonstrating how to calculate Renyi entropy using the bitermplus library.

```python
import bitermplus as btm
>>> # Preprocessing step
>>> # ...
>>> # Model fitting step
>>> # model = ...
>>> # Entropy calculation
>>> entropy = btm.entropy(model.matrix_topics_words_)

```

--------------------------------

### Example for get_top_topic_words

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Example demonstrating how to select top topic words from a fitted BTM model.

```python
>>> stable_topics = [0, 3, 10, 12, 18, 21]
>>> top_words = btm.get_top_topic_words(
...     model,
...     words_num=100,
...     topics_idx=stable_topics)

```

--------------------------------

### Perplexity Calculation Example

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.metrics.html

Example demonstrating how to calculate perplexity using the bitermplus library.

```python
import bitermplus as btm
>>> # Preprocessing step
>>> # ...
>>> # X, vocabulary, vocab_dict = btm.get_words_freqs(texts)
>>> # Model fitting step
>>> # model = ...
>>> # Inference step
>>> # p_zd = model.transform(docs_vec_subset)
>>> # Coherence calculation
>>> perplexity = btm.perplexity(model.matrix_topics_words_, p_zd, X, 8)

```

--------------------------------

### Vectorizer Setup

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

Initializes the CountVectorizer with default parameters, allowing for overrides via vectorizer_params.

```python
def _setup_vectorizer(self):
        """Initialize the vectorizer with default parameters."""
        default_params = {
            "lowercase": True,
            "token_pattern": r"\b[a-zA-Z][a-zA-Z0-9]*\b",
            "min_df": 1,
            "max_df": 0.95,
            "stop_words": "english",
        }
        default_params.update(self.vectorizer_params or {})
        return CountVectorizer(**default_params)
```

--------------------------------

### BTM Model Example

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.html

Demonstrates how to prepare data, create, fit, and transform a Biterm Topic Model using the BTM class.

```python
>>> import bitermplus as btm
>>> import pandas as pd
>>> from sklearn.feature_extraction.text import CountVectorizer
>>> 
>>> # Prepare data
>>> texts = ["machine learning is great", "I love deep learning"]
>>> vectorizer = CountVectorizer()
>>> X = vectorizer.fit_transform(texts)
>>> vocabulary = vectorizer.get_feature_names_out()
>>> 
>>> # Create and fit model
>>> model = btm.BTM(X, vocabulary, T=2, seed=42)
>>> docs_vec = btm.get_vectorized_docs(texts, vocabulary)
>>> biterms = btm.get_biterms(docs_vec)
>>> model.fit(biterms, iterations=100)
>>> 
>>> # Get results
>>> doc_topics = model.transform(docs_vec)
>>> print("Topics per document:", doc_topics.shape)
```

--------------------------------

### Coherence Calculation Example

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.metrics.html

Example demonstrating how to calculate semantic topic coherence using the bitermplus library.

```python
import bitermplus as btm
>>> # Preprocessing step
>>> # ...
>>> # X, vocabulary, vocab_dict = btm.get_words_freqs(texts)
>>> # Model fitting step
>>> # model = ...
>>> # Coherence calculation
>>> coherence = btm.coherence(model.matrix_topics_words_, X, M=20)

```

--------------------------------

### Transforming Documents

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.html

Example of transforming documents to topic probability distributions using the trained BTM model.

```python
>>> # Assuming model is fitted and docs_vec is prepared
>>> doc_topics = model.transform(docs_vec)
>>> print(f"Shape: {doc_topics.shape}")
>>> print(f"Topic distribution for first doc: {doc_topics[0]}")
```

```python
>>> # Using different inference types
>>> topics_biterm = model.transform(docs_vec, infer_type='sum_b')
>>> topics_word = model.transform(docs_vec, infer_type='sum_w')
```

--------------------------------

### Quick Start with BTMClassifier

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Demonstrates how to quickly use the BTMClassifier for topic modeling, including fitting the model and retrieving topic distributions and words.

```python
import bitermplus as btm

# Sample documents
texts = [
    "machine learning algorithms are powerful",
    "deep learning neural networks process data",
    "natural language processing understands text",
    "artificial intelligence transforms industries"
]

# Create and fit model (one step!)
model = btm.BTMClassifier(n_topics=2, random_state=42)
model.fit(texts)

# Get topic distributions
doc_topics = model.transform(texts)
print(f"Document-topic matrix shape: {doc_topics.shape}")

# Interpret topics
topic_words = model.get_topic_words(n_words=5)
for topic_id, words in topic_words.items():
    print(f"Topic {topic_id}: {', '.join(words)}")
```

--------------------------------

### Grid Search with BTMClassifier

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Example of using scikit-learn's GridSearchCV to find optimal hyperparameters for the BTMClassifier.

```python
from sklearn.model_selection import GridSearchCV

param_grid = {
    'n_topics': [3, 5, 8],
    'alpha': [0.1, 0.5, 1.0],
    'max_iter': [100, 300]
}

grid_search = GridSearchCV(
    btm.BTMClassifier(random_state=42),
    param_grid,
    cv=3,
    scoring=None  # Uses model's score method
)

grid_search.fit(texts)
best_model = grid_search.best_estimator_
```

--------------------------------

### Model Fitting

Source: https://bitermplus.readthedocs.io/en/latest/tutorial.html

Demonstrates basic model fitting, including data import, document vectorization, biterm extraction, and model initialization and fitting.

```python
import bitermplus as btm
import numpy as np
import pandas as pd

# Importing data
df = pd.read_csv(
    'dataset/SearchSnippets.txt.gz', header=None, names=['texts'])
texts = df['texts'].str.strip().tolist()

# Vectorize documents and extract biterms
# Uses sklearn's CountVectorizer internally - accepts its parameters
# Example: stop word removal
stop_words = ["word1", "word2", "word3"]
X, vocabulary, vocab_dict = btm.get_words_freqs(texts, stop_words=stop_words)
docs_vec = btm.get_vectorized_docs(texts, vocabulary)
biterms = btm.get_biterms(docs_vec)

# Initializing and running model
model = btm.BTM(
    X, vocabulary, seed=12321, T=8, M=20, alpha=50/8, beta=0.01)
model.fit(biterms, iterations=20)

```

--------------------------------

### Model Loading and Saving

Source: https://bitermplus.readthedocs.io/en/latest/tutorial.html

Demonstrates model serialization using pickle for saving and loading models.

```python
import pickle as pkl
# Saving
with open("model.pkl", "wb") as file:
    pkl.dump(model, file)

# Loading
with open("model.pkl", "rb") as file:
    model = pkl.load(file)

```

--------------------------------

### Model Loading and Saving

Source: https://bitermplus.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Shows how to save and load trained BitermPlus models using Python's `pickle` module.

```python
import pickle as pkl
# Saving
with open("model.pkl", "wb") as file:
    pkl.dump(model, file)

# Loading
with open("model.pkl", "rb") as file:
    model = pkl.load(file)
```

--------------------------------

### Calculating Metrics

Source: https://bitermplus.readthedocs.io/en/latest/tutorial.html

Calculates perplexity and coherence using the document-topic probability matrix and model attributes.

```python
perplexity = btm.perplexity(model.matrix_topics_words_, p_zd, X, 8)
coherence = btm.coherence(model.matrix_topics_words_, X, M=20)
# or
perplexity = model.perplexity_
coherence = model.coherence_

```

--------------------------------

### Inference

Source: https://bitermplus.readthedocs.io/en/latest/_sources/tutorial.rst.txt

Shows how to calculate the document-topic probability matrix (inference) using the fitted model. It also explains how to vectorize new documents for inference using the training vocabulary.

```python
p_zd = model.transform(docs_vec)
```

```python
new_docs_vec = btm.get_vectorized_docs(new_texts, vocabulary)
p_zd = model.transform(new_docs_vec)
```

--------------------------------

### Inference

Source: https://bitermplus.readthedocs.io/en/latest/tutorial.html

Calculates the document-topic probability matrix for inference.

```python
p_zd = model.transform(docs_vec)

```

```python
new_docs_vec = btm.get_vectorized_docs(new_texts, vocabulary)
p_zd = model.transform(new_docs_vec)

```

--------------------------------

### BTM Model Initialization and Fitting

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

This snippet shows the initialization and fitting process of the BTM model within the `fit` method.

```python
        biterms = get_biterms(docs_vec, win=self.window_size)

        # Adjust coherence window to not exceed vocabulary size
        effective_coherence_window = min(self.coherence_window, len(vocabulary))

        # Initialize and fit BTM model
        self.model_ = BTM(
            doc_term_matrix,
            vocabulary,
            T=self.n_topics,
            M=effective_coherence_window,
            alpha=self.alpha,
            beta=self.beta,
            seed=self.random_state or 0,
            win=self.window_size,
            has_background=self.has_background,
            epsilon=self.epsilon,
        )

        self.model_.fit(biterms, iterations=self.max_iter, verbose=verbose)

        return self
```

--------------------------------

### Get Top Words for Topics

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

The `get_topic_words` method retrieves the most relevant words for specified topics.

```python
    def get_topic_words(
        self, topic_id: Optional[int] = None, n_words: int = 10
    ) -> Union[List[str], Dict[int, List[str]]]:
        """Get top words for topics.

        Parameters
        ----------
        topic_id : int, optional
            If provided, return words for this topic only.
            If None, return words for all topics.
        n_words : int, default=10
            Number of top words to return per topic.

        Returns
        -------
        topic_words : list or dict
            If topic_id is provided, returns list of top words for that topic.
            Otherwise, returns dict mapping topic_id to list of words.
        """
        check_is_fitted(self, "model_")

        topic_word_matrix = self.model_.matrix_topics_words_

        if topic_id is not None:
            if not 0 <= topic_id < self.n_topics:
                raise ValueError(f"topic_id must be between 0 and {self.n_topics - 1}")
            word_indices = np.argsort(topic_word_matrix[topic_id])[-n_words:][::-1]
            return self.vocabulary_[word_indices].tolist()
        else:
            result = {}
            for t in range(self.n_topics):
                word_indices = np.argsort(topic_word_matrix[t])[-n_words:][::-1]
                result[t] = self.vocabulary_[word_indices].tolist()
            return result
```

--------------------------------

### Cross-validation with BTMClassifier

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Example of using scikit-learn's cross_val_score with the BTMClassifier for evaluating model performance.

```python
from sklearn.model_selection import cross_val_score

model = btm.BTMClassifier(n_topics=5, random_state=42)
scores = cross_val_score(model, texts, cv=3)
print(f"Mean coherence: {scores.mean():.3f}")
```

--------------------------------

### BTMClassifier transform method

Source: https://bitermplus.readthedocs.io/en/latest/sklearn_api.html

Example of using the transform method of the BTMClassifier class to transform documents into topic distributions.

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from bitermplus import BTMClassifier

X = ["This is the first document.", "This document is the second document.", "And this is the third one.", "Is this the first document?"]

vectorizer = TfidfVectorizer(stop_words='english')
X_vec = vectorizer.fit_transform(X)

b = BTMClassifier(n_topics=2, random_state=0, verbose=False)
b.fit(X_vec)

doc_topic_matrix = b.transform(X_vec)
print(doc_topic_matrix.shape)

```

--------------------------------

### Complete preprocessing pipeline

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Illustrates a complete preprocessing pipeline, including text vectorization, biterm generation, and BTM model initialization and fitting.

```python
>>> texts = ["AI and machine learning", "Natural language processing"]
>>> X, vocabulary, vocab_dict = btm.get_words_freqs(texts)
>>> docs_vec = btm.get_vectorized_docs(texts, vocabulary)
>>> biterms = btm.get_biterms(docs_vec, win=15)
>>> # Now ready for BTM training
>>> model = btm.BTM(X, vocabulary, T=2)
>>> model.fit(biterms)

```

--------------------------------

### Get Dominant Topics for Documents

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

The `get_document_topics` method identifies the dominant topics for a given set of documents based on a probability threshold.

```python
    def get_document_topics(
        self, X: Union[List[str], pd.Series], threshold: float = 0.1
    ) -> List[List[int]]:
        """Get dominant topics for documents.

        Parameters
        ----------
        X : array-like of shape (n_documents,)
            Documents to analyze.
        threshold : float, default=0.1
            Minimum probability threshold for topic assignment.

        Returns
        -------
        doc_topics : list of list of int
            For each document, list of topic IDs above threshold.
        """
        doc_topic_probs = self.transform(X)
        doc_topics = []

        for doc_probs in doc_topic_probs:
            topics = [i for i, prob in enumerate(doc_probs) if prob >= threshold]
            doc_topics.append(topics)

        return doc_topics
```

--------------------------------

### Custom Preprocessing with vectorizer_params

Source: https://bitermplus.readthedocs.io/en/latest/sklearn_api.html

Example of customizing the text preprocessing by passing parameters to the internal CountVectorizer via the `vectorizer_params` argument of BTMClassifier.

```python
custom_params = {
    'min_df': 2,           # Minimum document frequency
    'max_df': 0.8,         # Maximum document frequency
    'stop_words': 'english',  # Remove English stop words
    'lowercase': True,     # Convert to lowercase
    'token_pattern': r'\b[a-zA-Z]{3,}\b'  # Only words 3+ chars
}

model = btm.BTMClassifier(
    n_topics=5,
    vectorizer_params=custom_params
)


```

--------------------------------

### Visualizing Results

Source: https://bitermplus.readthedocs.io/en/latest/tutorial.html

Visualizes results using the tmplot package's interactive report interface.

```python
import tmplot as tmp

# Run the interactive report interface
tmp.report(model=model, docs=texts)

```

--------------------------------

### Complete preprocessing pipeline

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Demonstrates a complete preprocessing pipeline using get_words_freqs, get_vectorized_docs, and get_biterms.

```python
texts = ["AI and ML are exciting", "Deep learning transforms data"]
X, vocabulary, vocab_dict = btm.get_words_freqs(texts)
docs_vectorized = btm.get_vectorized_docs(texts, vocabulary)
biterms = btm.get_biterms(docs_vectorized)
```

--------------------------------

### Filtering Stable Topics

Source: https://bitermplus.readthedocs.io/en/latest/tutorial.html

Identifies stable topics using distance metrics provided by the tmplot package, loading saved models and calculating close and stable topics.

```python
import pickle as pkl
import tmplot as tmp
import glob

# Loading saved models
models_files = sorted(glob.glob(r'results/model[0-9].pkl'))
models = []
for fn in models_files:
    file = open(fn, 'rb')
    models.append(pkl.load(file))
    file.close()

# Choosing reference model
np.random.seed(122334)
reference_model = np.random.randint(1, 6)

# Getting close topics
close_topics, close_kl = tmp.get_closest_topics(
    models, method="sklb", ref=reference_model)

# Getting stable topics
stable_topics, stable_kl = tmp.get_stable_topics(
    close_topics, close_kl, ref=reference_model, thres=0.7)

# Stable topics indices list
print(stable_topics[:, reference_model])

```

--------------------------------

### BTMClassifier Initialization

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

Initializes the BTMClassifier with various parameters controlling the topic modeling process.

```python
def __init__(
        self,
        n_topics: int = 8,
        alpha: Optional[float] = None,
        beta: float = 0.01,
        max_iter: int = 600,
        random_state: Optional[int] = None,
        window_size: int = 15,
        has_background: bool = False,
        coherence_window: int = 20,
        vectorizer_params: Optional[Dict[str, Any]] = None,
        epsilon: float = 1e-10,
    ):
        self.n_topics = n_topics
        self.beta = beta
        self.max_iter = max_iter
        self.random_state = random_state
        self.window_size = window_size
        self.has_background = has_background
        self.coherence_window = coherence_window
        self.vectorizer_params = vectorizer_params
        self.epsilon = epsilon

        # Validate parameters before calculating alpha
        self._validate_params()
        self.alpha = alpha if alpha is not None else 50.0 / n_topics

        # Validate alpha after calculation
        if self.alpha <= 0:
            raise ValueError("alpha must be positive")
```

--------------------------------

### Basic usage of get_vectorized_docs

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Illustrates the basic usage of the get_vectorized_docs function to convert text documents into a vectorized representation using word IDs.

```python
import bitermplus as btm
texts = ["machine learning is great", "I love deep learning"]
X, vocabulary, _ = btm.get_words_freqs(texts)
docs_vec = btm.get_vectorized_docs(texts, vocabulary)
print(f"Original: {texts[0]}")
print(f"Vectorized: {docs_vec[0]}")
```

--------------------------------

### Basic usage of get_words_freqs

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Demonstrates how to use the get_words_freqs function to extract word frequencies and vocabulary from a list of text documents.

```python
import bitermplus as btm
texts = ["machine learning is great", "I love natural language processing"]
X, vocabulary, vocab_dict = btm.get_words_freqs(texts)
print(f"Matrix shape: {X.shape}")
print(f"Vocabulary size: {len(vocabulary)}")
```

--------------------------------

### Choosing Different Inference Methods

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Illustrates how to select different inference methods ('sum_b', 'sum_w', 'mix') when transforming new texts.

```python
model = btm.BTMClassifier(n_topics=5)
model.fit(texts)

# Different inference types
topics_sum_b = model.transform(new_texts, infer_type='sum_b')  # Default
topics_sum_w = model.transform(new_texts, infer_type='sum_w')  # Word-based
topics_mix = model.transform(new_texts, infer_type='mix')      # Mixed
```

--------------------------------

### With custom window size

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Shows how to use the get_biterms function with a custom window size.

```python
>>> biterms = btm.get_biterms(docs_vec, win=10)

```

--------------------------------

### get_top_topic_docs function

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_util.html

Select top topic docs from a fitted model.

```Python
def get_top_topic_docs(
    docs: Sequence[Any],
    p_zd: np.ndarray,
    docs_num: int = 20,
    topics_idx: Sequence[Any] = None
) -> DataFrame:
    """Select top topic docs from a fitted model.

    Parameters
    ----------
    docs : Sequence[Any]
        Iterable of documents (e.g. list of strings).
    p_zd : np.ndarray
        Documents vs topics probabilities matrix.
    docs_num : int = 20
        The number of documents to select.
    topics_idx : Sequence[Any] = None
        Topics indices. Meant to be used to select only stable
        topics.

    Returns
    -------
    DataFrame
        Documents with highest probabilities in all selected topics.

    Example
    -------
    >>> top_docs = btm.get_top_topic_docs(
    ...     texts,
    ...     p_zd,
    ...     docs_num=100,
    ...     topics_idx=[1,2,3,4])
    """

    def _select_docs(docs, p_zd, topic_id: int):
        probs = p_zd[:, topic_id]
        idx = np.argsort(probs)[: -docs_num - 1 : -1]
        result = Series(np.asarray(docs)[idx])
        result.name = "topic{}".format(topic_id)
        return result

    topics_num = p_zd.shape[1]
    topics_idx = np.arange(topics_num) if topics_idx is None else topics_idx
    return concat(map(lambda x: _select_docs(docs, p_zd, x), topics_idx), axis=1)
```

--------------------------------

### Basic usage

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Demonstrates the basic usage of the get_biterms function to extract biterms from a list of texts.

```python
>>> import bitermplus as btm
>>> texts = ["machine learning algorithms", "deep learning networks"]
>>> X, vocabulary, _ = btm.get_words_freqs(texts)
>>> docs_vec = btm.get_vectorized_docs(texts, vocabulary)
>>> biterms = btm.get_biterms(docs_vec)
>>> print(f"Number of documents: {len(biterms)}")
>>> print(f"Biterms in first doc: {biterms[0]}")

```

--------------------------------

### Fit and Transform Documents

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

The `fit_transform` method combines fitting the model and transforming documents in a single step.

```python
    def fit_transform(
        self, X: Union[List[str], pd.Series], y=None, infer_type: str = "sum_b",
        verbose: bool = False,
    ) -> np.ndarray:
        """Fit model and transform documents in one step.

        Parameters
        ----------
        X : array-like of shape (n_documents,)
            Documents to fit and transform.
        y : Ignored
            Not used, present for sklearn compatibility.
        infer_type : str, default='sum_b'
            Inference method. Options: 'sum_b', 'sum_w', 'mix'.
        verbose : bool, default=False
            Whether to show a progress bar during training.

        Returns
        -------
        doc_topic_matrix : np.ndarray of shape (n_documents, n_topics)
            Document-topic probability matrix.
        """
        return self.fit(X, verbose=verbose).transform(X, infer_type=infer_type)
```

--------------------------------

### API Comparison: Traditional vs. Sklearn API

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Compares the traditional multi-step API with the new simplified sklearn-style API for BitermPlus.

```python
# Multiple manual preprocessing steps
X, vocabulary, vocab_dict = btm.get_words_freqs(texts)
docs_vec = btm.get_vectorized_docs(texts, vocabulary)
biterms = btm.get_biterms(docs_vec)

# Model creation and fitting
model = btm.BTM(X, vocabulary, seed=42, T=3, M=20, alpha=50/3, beta=0.01)
model.fit(biterms, iterations=100)

# Inference
p_zd = model.transform(docs_vec)
```

```python
# Everything in one step!
model = btm.BTMClassifier(n_topics=3, random_state=42)
doc_topics = model.fit_transform(texts)
```

--------------------------------

### API Comparison - Traditional API

Source: https://bitermplus.readthedocs.io/en/latest/sklearn_api.html

Illustrates the multi-step process of the traditional BitermPlus API for topic modeling.

```python
# Multiple manual preprocessing steps
X, vocabulary, vocab_dict = btm.get_words_freqs(texts)
docs_vec = btm.get_vectorized_docs(texts, vocabulary)
biterms = btm.get_biterms(docs_vec)

# Model creation and fitting
model = btm.BTM(X, vocabulary, seed=42, T=3, M=20, alpha=50/3, beta=0.01)
model.fit(biterms, iterations=100)

# Inference
p_zd = model.transform(docs_vec)

```

--------------------------------

### Custom Preprocessing Parameters

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Demonstrates how to control text preprocessing using the `vectorizer_params` argument in `BTMClassifier`.

```python
custom_params = {
    'min_df': 2,           # Minimum document frequency
    'max_df': 0.8,         # Maximum document frequency
    'stop_words': 'english',  # Remove English stop words
    'lowercase': True,     # Convert to lowercase
    'token_pattern': r'\b[a-zA-Z]{3,}\b'  # Only words 3+ chars
}

model = btm.BTMClassifier(
    n_topics=5,
    vectorizer_params=custom_params
)
```

--------------------------------

### Basic usage

Source: https://bitermplus.readthedocs.io/en/latest/sklearn_api.html

Demonstrates the fundamental steps of initializing, fitting, and transforming text data using the BTMClassifier.

```python
import bitermplus as btm
texts = [
    "machine learning algorithms are powerful",
    "deep learning neural networks process data",
    "natural language processing understands text"
]
model = btm.BTMClassifier(n_topics=2, random_state=42)
model.fit(texts)
print(f"Shape: {doc_topics.shape}")
```

--------------------------------

### get_words_freqs with custom parameters

Source: https://bitermplus.readthedocs.io/en/latest/bitermplus.util.html

Shows how to use get_words_freqs with custom parameters like min_df, stop_words, and lowercase.

```python
X, vocab, vocab_dict = btm.get_words_freqs(
    texts, min_df=1, stop_words='english', lowercase=True
)
```

--------------------------------

### Migration from Original API to Sklearn-style API

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Compares the old `bitermplus` API usage with the new `BTMClassifier` sklearn-style API, highlighting the simplified preprocessing.

```python
# Original bitermplus API
X, vocabulary, vocab_dict = btm.get_words_freqs(texts)
docs_vec = btm.get_vectorized_docs(texts, vocabulary)
biterms = btm.get_biterms(docs_vec)

model = btm.BTM(X, vocabulary, seed=42, T=8, M=20, alpha=50/8, beta=0.01)
model.fit(biterms, iterations=600)
p_zd = model.transform(docs_vec)
```

```python
# New sklearn-style API
model = btm.BTMClassifier(
    n_topics=8,
    random_state=42,
    coherence_window=20,
    alpha=50/8,
    beta=0.01,
    max_iter=600,
    epsilon=1e-10  # Numerical stability parameter
)
p_zd = model.fit_transform(texts)

The new API handles all preprocessing automatically while providing the same underlying functionality with much simpler usage.
```

--------------------------------

### Model Evaluation Metrics

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Shows how to calculate and print topic coherence scores, mean coherence, and perplexity for the trained model.

```python
model = btm.BTMClassifier(n_topics=5, random_state=42)
model.fit(texts)

# Coherence per topic
coherence_scores = model.coherence_
print(f"Topic coherence: {coherence_scores}")

# Overall model quality
mean_coherence = model.score(texts)
print(f"Mean coherence: {mean_coherence:.3f}")

# Perplexity (lower is better)
model.transform(texts)  # Required for perplexity calculation
perplexity = model.perplexity_
print(f"Perplexity: {perplexity:.3f}")
```

--------------------------------

### Transform Documents to Topic Distribution

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_api.html

The `transform` method converts documents into their topic distributions using the fitted BTM model.

```python
    def transform(
        self, X: Union[List[str], pd.Series], infer_type: str = "sum_b"
    ) -> np.ndarray:
        """Transform documents to topic distribution.

        Parameters
        ----------
        X : array-like of shape (n_documents,)
            Documents to transform.
        infer_type : str, default='sum_b'
            Inference method. Options: 'sum_b', 'sum_w', 'mix'.

        Returns
        -------
        doc_topic_matrix : np.ndarray of shape (n_documents, n_topics)
            Document-topic probability matrix.
        """
        check_is_fitted(self, "model_")

        # Convert input to list of strings
        if isinstance(X, pd.Series):
            X = X.tolist()
        elif not isinstance(X, list):
            X = list(X)

        # Vectorize documents using the fitted vectorizer's analyzer
        docs_vec = self._get_vectorized_docs(X)

        # Transform using BTM model
        return self.model_.transform(docs_vec, infer_type=infer_type, verbose=False)
```

--------------------------------

### Pipeline Integration with BTMClassifier

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Demonstrates integrating BTMClassifier into a scikit-learn Pipeline, including text preprocessing.

```python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer

def preprocess_text(texts):
    return [text.lower().replace(',', '') for text in texts]

pipeline = Pipeline([
    ('preprocess', FunctionTransformer(preprocess_text)),
    ('btm', btm.BTMClassifier(n_topics=3, random_state=42))
])

doc_topics = pipeline.fit_transform(texts)
```

--------------------------------

### Working with Pandas DataFrames

Source: https://bitermplus.readthedocs.io/en/latest/_sources/sklearn_api.rst.txt

Demonstrates seamless integration with pandas DataFrames, including fitting the model and adding topic predictions to the DataFrame.

```python
import pandas as pd

df = pd.DataFrame({'text': texts, 'category': ['ML', 'DL', 'NLP', 'AI']})

model = btm.BTMClassifier(n_topics=3)
doc_topics = model.fit_transform(df['text'])

# Add topic predictions to DataFrame
df['dominant_topic'] = doc_topics.argmax(axis=1)
df['topic_confidence'] = doc_topics.max(axis=1)
```

--------------------------------

### _select_words function

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_util.html

Internal helper function to select top words for a given topic.

```Python
def _select_words(model, topic_id: int):
    probs = model.matrix_topics_words_[topic_id, :]
    idx = np.argsort(probs)[: -words_num - 1 : -1]
    result = Series(model.vocabulary_[idx])
    result.name = "topic{}".format(topic_id)
    return result

topics_num = model.topics_num_
topics_idx = np.arange(topics_num) if topics_idx is None else topics_idx
return concat(map(lambda x: _select_words(model, x), topics_idx), axis=1)
```

--------------------------------

### get_docs_top_topic function

Source: https://bitermplus.readthedocs.io/en/latest/_modules/bitermplus/_util.html

Select most probable topic for each document.

```Python
def get_docs_top_topic(docs: Sequence[Any], p_zd: np.ndarray) -> DataFrame:
    """Select most probable topic for each document.

    Parameters
    ----------
    docs : Sequence[Any]
        Iterable of documents (e.g. list of strings).
    p_zd : np.ndarray
        Documents vs topics probabilities matrix.

    Returns
    -------
    DataFrame
        Documents and the most probable topic for each of them.

    Example
    -------
    >>> import bitermplus as btm
    >>> # Read documents from file
    >>> # texts = ...
    >>> # Build and train a model
    >>> # model = ...
    >>> # model.fit(...)
    >>> btm.get_docs_top_topic(texts, model.matrix_docs_topics_)
    """
    return DataFrame({"documents": docs, "label": p_zd.argmax(axis=1)})
```

--------------------------------

### API Comparison - New Sklearn API

Source: https://bitermplus.readthedocs.io/en/latest/sklearn_api.html

Shows the simplified, one-step process using the new scikit-learn compatible BTMClassifier.

```python
# Everything in one step!
model = btm.BTMClassifier(n_topics=3, random_state=42)
doc_topics = model.fit_transform(texts)

```