### Install WizMap Node.js Dependencies

Source: https://github.com/poloclub/wizmap/blob/main/notebook-widget/README.md

This command installs the necessary Node.js dependencies for the WizMap project using npm. Ensure Node.js and npm are installed on your system.

```bash
npm install
```

--------------------------------

### Install WizMap Python Library

Source: https://github.com/poloclub/wizmap/blob/main/notebook-widget/README.md

Installs the Wizmap Python library using pip. This is the first step to use WizMap with your own embeddings.

```bash
pip install wizmap
```

--------------------------------

### Run WizMap Development Server

Source: https://github.com/poloclub/wizmap/blob/main/notebook-widget/README.md

This command starts the development server for WizMap using npm. After running, you can access the application by navigating to http://localhost:3000 in your browser.

```bash
npm run dev
```

--------------------------------

### Install WizMap and Dependencies

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

Installs the WizMap library along with umap-learn, which is required for dimensionality reduction. This is a necessary first step before using WizMap functionalities.

```python
# Install wizmap
# !pip install --upgrade wizmap umap-learn
```

--------------------------------

### Example WizMap Sharing URL

Source: https://github.com/poloclub/wizmap/blob/main/notebook-widget/README.md

An example URL for sharing a WizMap visualization of IMDB embeddings. This URL points to the data and grid JSON files hosted on Hugging Face.

```url
https://poloclub.github.io/wizmap/?dataURL=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Fxiaohk%2Fembeddings%2Fresolve%2Fmain%2Fimdb%2Fdata.ndjson&gridURL=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Fxiaohk%2Fembeddings%2Fresolve%2Fmain%2Fimdb%2Fgrid.json
```

--------------------------------

### Clone WizMap Repository

Source: https://github.com/poloclub/wizmap/blob/main/notebook-widget/README.md

This command clones the WizMap project repository from GitHub. It's the initial step to get the project code onto your local machine.

```bash
git clone git@github.com:poloclub/wizmap.git
```

--------------------------------

### Calculate and Reshape Log Density

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This snippet computes the log-likelihood density for each point in the 'grid' using the fitted KDE model. It then exponentiates these values to get the probability density and reshapes the result into a 2D grid matching the 'xx' shape.

```python
# Sklearn
log_density = kde.score_samples(grid)
log_density = np.exp(log_density)
grid_density = np.reshape(log_density, xx.shape)
grid_density.shape
```

--------------------------------

### Load Pre-extracted Embeddings

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

Loads pre-extracted embeddings from a remote URL for demonstration purposes, saving the time required for OpenAI API calls. This bypasses the need for an API key and local data files for this specific example.

```python
# To save time, we will just load pre-extracted embeddings
EMBEDDING_URL = "https://huggingface.co/datasets/xiaohk/embeddings/resolve/main/diffusiondb/openai-text-embeddings.npz"
stream = requests.get(EMBEDDING_URL, stream=True)
embedding_data = np.load(BytesIO(stream.content), allow_pickle=True)
```

--------------------------------

### Generate JSON for WizMap Data

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

This code snippet is a placeholder for generating the second JSON file required by WizMap, which encodes the raw data. In this specific example, the text content of the JSON is represented as a string.

```python
# Here we are writing a json string as the text content

```

--------------------------------

### Get Top N Indices from Sparse Matrix (Python)

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This Python function efficiently retrieves the indices of the top N values for each row in a sparse CSR matrix. It utilizes `np.argpartition` for fast selection and handles cases where a row has fewer than N non-zero elements. The function returns a NumPy array where each row contains the top N indices for the corresponding row in the input matrix.

```python
from scipy.sparse import csr_matrix
import numpy as np

def top_n_idx_sparse(matrix: csr_matrix, n: int) -> np.ndarray:
    """ Return indices of top n values in each row of a sparse matrix
    Retrieved from:
        https://github.com/MaartenGr/BERTopic/blob/master/bertopic/_bertopic.py#L2801
    Arguments:
        matrix: The sparse matrix from which to get the top n indices per row
        n: The number of highest values to extract from each row
    Returns:
        indices: The top n indices per row
    """
    indices = []
    for le, ri in zip(matrix.indptr[:-1], matrix.indptr[1:]):
        n_row_pick = min(n, ri - le)
        values = matrix.indices[le + np.argpartition(matrix.data[le:ri], -n_row_pick)[-n_row_pick:]]
        values = [values[index] if len(values) >= index + 1 else None for index in range(n)]
        indices.append(values)
    return np.array(indices)
```

--------------------------------

### Get Top N Values from Sparse Matrix (Python)

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This Python function retrieves the top N values (scores) for each row in a sparse CSR matrix, given the indices of those top values. It iterates through the provided indices and extracts the corresponding matrix values, returning them as a NumPy array. It handles cases where an index might be `None` by assigning a score of 0.

```python
from scipy.sparse import csr_matrix
import numpy as np

def top_n_values_sparse(matrix: csr_matrix, indices: np.ndarray) -> np.ndarray:
    """ Return the top n values for each row in a sparse matrix
    Arguments:
        matrix: The sparse matrix from which to get the top n indices per row
        indices: The top n indices per row
    Returns:
        top_values: The top n scores per row
    """
    top_values = []
    for row in range(indices.shape[0]):
        scores = np.array([matrix[row, c] if c is not None else 0 for c in indices[row, :]])
        top_values.append(scores)
    return np.array(top_values)
```

--------------------------------

### Initialize Quadtree and Count Matrix

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Demonstrates the initialization of a quadtree and the construction of a count matrix from text data. It first builds the quadtree by adding data, then obtains the root node representation. Subsequently, it uses `CountVectorizer` to create a count matrix and extract n-gram feature names.

```python
# Build the quadtree
tree = Quadtree()
tree.add_all_data(data)

# Build the count matrix
root = tree.get_node_representation()

cv = CountVectorizer(stop_words="english", ngram_range=(1, 1))
count_mat = cv.fit_transform(texts)
ngrams = cv.get_feature_names_out()
```

--------------------------------

### Define Working and Data Directories - Python

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Sets up the primary working directory and data directory paths for the project. These paths are crucial for organizing input data and storing intermediate or final results.

```python
WORK_DIR = "/nvmescratch/jay/wizmap/temp/"
DATA_DIR = "/nvmescratch/jay/wizmap/data/"
```

--------------------------------

### Generate KDE for Each Year

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This loop iterates through a range of years (1980-2022) and computes a Kernel Density Estimation for each year's data subset. It fits a KDE model, calculates the density on the predefined grid, and stores the resulting grid density in the 'time_grids' dictionary.

```python
time_grids = {}
for cur_year in tqdm(range(1980, 2022)):
    cur_kde = KernelDensity(kernel='gaussian', bandwidth=bw)

    cur_indexes = np.where(umap_df['years'] == cur_year)[0]
    cur_kde.fit(projected_emb[cur_indexes, :])

    # Compute KDE and transform it to likelihood scale
    cur_log_density = cur_kde.score_samples(grid)
    cur_density = np.exp(cur_log_density)
    cur_grid_density = np.reshape(cur_density, xx.shape)
    cur_grid = cur_grid_density.astype(float).round(4).tolist()

    # Record this year's grid
    time_grids[str(cur_year)] = cur_grid
```

--------------------------------

### Initialize Sentence Transformer Model - Python

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Initializes a Sentence Transformer model ('all-mpnet-base-v2') on the appropriate device (CUDA if available, otherwise CPU). This model is used for generating embeddings from text.

```python
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

model = SentenceTransformer('all-mpnet-base-v2', device=device)
```

--------------------------------

### Compute KDE Bandwidth using Silverman's Rule

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This snippet calculates the bandwidth for a Gaussian Kernel Density Estimation (KDE) using Silverman's rule of thumb. It's designed for a given sample size and dimensionality, aiming for an optimal bandwidth for density estimation.

```python
# Compute the bandwidth using silverman's rule
sample_size = 100000
n = sample_size
d = projected_emb.shape[1]
bw = (n * (d + 2) / 4.)**(-1. / (d + 4))
```

--------------------------------

### Set Visualization Parameters

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Sets up parameters for visualization, including the maximum zoom scale and SVG dimensions. It also calculates the data domain (min/max x and y values) from the provided data points, which are essential for scaling and positioning elements within the visualization.

```python
max_zoom_scale = 20
svg_width = 800
svg_height = 800
xs = [d['x'] for d in data]
ys = [d['y'] for d in data]
x_domain = [np.min(xs), np.max(xs)]
y_domain = [np.min(ys), np.max(ys)]


```

--------------------------------

### Import Libraries for ACL Metadata Analysis - Python

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Imports essential Python libraries for data manipulation, natural language processing, machine learning, and visualization, specifically for analyzing ACL metadata.

```python
from glob import glob
from os.path import exists, join, basename
from tqdm import tqdm
from json import load, dump
from matplotlib import pyplot as plt
from collections import Counter
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from quadtreed3 import Quadtree, Node
from scipy.sparse import csr_matrix
from sklearn.neighbors import KernelDensity
from scipy.stats import norm
from typing import Tuple

import re
import os
import shutil
import random
import cuml
import pickle
import torch
import ndjson

import pandas as pd
import numpy as np

SEED = 20220101

# plt.style.use('ggplot')
# plt.rcParams['figure.dpi'] = 300
```

--------------------------------

### Visualize Embeddings with WizMap in Notebook

Source: https://github.com/poloclub/wizmap/blob/main/notebook-widget/README.md

Demonstrates how to visualize embedding data using the wizmap.visualize() function within a computational notebook environment. This function takes pre-computed JSON files containing embedding summaries, distributions, and the original embedding data as input.

```python
import wizmap

# Assuming json_data_url and json_grid_url are URLs to your pre-computed JSON files
data_url = "YOUR_DATA_URL"
grid_url = "YOUR_GRID_URL"

wizmap.visualize(dataURL=data_url, gridURL=grid_url)
```

--------------------------------

### Prepare and Save Density Data to JSON

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This code compiles various density-related data into a JSON-compatible dictionary, including the main grid density, x/y ranges, sample sizes, and yearly density grids. It then saves this dictionary to a JSON file named 'umap-grid.json'.

```python
x_min, x_max, y_min, y_max = float(x_min), float(x_max), float(y_min), float(y_max)

grid_density_json = {
    'grid': grid_density.astype(float).round(4).tolist(),
    'xRange': [x_min, x_max],
    'yRange': [y_min, y_max],
    'sampleSize': sample_size,
    'totalPointSize': umap_df.shape[0],
    'padded': True,
    'timeGrids': time_grids
}

dump(grid_density_json, open(join(DATA_DIR, 'umap-grid.json'), 'w'))
```

--------------------------------

### Fit Kernel Density Estimation Model

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This code fits a Gaussian Kernel Density Estimation model ('kde') to a random sample of the projected embeddings. It uses a pre-calculated bandwidth ('bw') and a random state for reproducibility.

```python
# We use a random sample to fit the KDE for faster run time
rng = np.random.RandomState(SEED)
random_indexes = rng.choice(range(projected_emb.shape[0]),
                            min(projected_emb.shape[0], sample_size),
                            replace=False)

kde = KernelDensity(kernel='gaussian', bandwidth=bw)
kde.fit(projected_emb[random_indexes, :])
```

--------------------------------

### Import Libraries for WizMap Visualization

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

Imports essential Python libraries for data manipulation, visualization, and machine learning, including wizmap, numpy, pandas, matplotlib, UMAP, and OpenAI's async client. These are used throughout the project for data processing and visualization.

```python
from glob import glob
from os.path import exists, join, basename
from tqdm import tqdm
from json import load, dump
from matplotlib import pyplot as plt
from collections import Counter

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from scipy.sparse import csr_matrix
from sklearn.neighbors import KernelDensity
from scipy.stats import norm
from typing import Tuple
from io import BytesIO
from umap import UMAP
from openai import AsyncOpenAI

import pandas as pd
import numpy as np
import json
import requests
import urllib
import wizmap

SEED = 20230501

plt.rcParams["figure.dpi"] = 300
wizmap.__version__
```

--------------------------------

### WizMap Citation in BibTeX Format

Source: https://github.com/poloclub/wizmap/blob/main/notebook-widget/README.md

BibTeX entry for citing the WizMap research paper, titled 'WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddings'.

```bibtex
@article{wangWizMapScalableInteractive2023,
  title = {{{WizMap}}}: {{Scalable Interactive Visualization}} for {{Exploring Large Machine Learning Embeddings}}},
  shorttitle = {{{WizMap}}},
  author = {Wang, Zijie J. and Hohman, Fred and Chau, Duen Horng},
  year = {2023},
  url = {http://arxiv.org/abs/2306.09328},
  urldate = {2023-06-16},
  archiveprefix = {arxiv},
  journal = {arXiv 2306.09328}
}
```

--------------------------------

### Initialize UMAP for Dimensionality Reduction - Python

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Initializes a UMAP (Uniform Manifold Approximation and Projection) reducer from the cuml library. It's configured with specific parameters for nearest neighbors, minimum distance, metric, and number of components for projecting high-dimensional embeddings into a 2D space.

```python
n_neighbors = 60
min_dist = 0.1
reducer_cuml = cuml.UMAP(
        n_neighbors=n_neighbors,
        min_dist=min_dist,
        metric='cosine',
        n_components=2,
        verbose=False,
        random_state=SEED
    )

# Fit UMAP
projected_emb_cuml = reducer_cuml.fit_transform(embeddings)
```

--------------------------------

### Display WizMap Visualization (Python)

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

This Python snippet demonstrates how to display a WizMap visualization using the previously defined `data_url` and `grid_url`. It utilizes a `wizmap.visualize` function, suggesting the need for a `wizmap` library or module. The `height` parameter allows customization of the visualization's display size.

```python
# Display wizmap
wizmap.visualize(data_url, grid_url, height=700)
```

--------------------------------

### Import Libraries for Data Visualization and Analysis

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

Imports essential Python libraries for data manipulation, natural language processing, dimensionality reduction, and plotting. These libraries are fundamental for the entire data processing and visualization pipeline.

```python
from glob import glob
from os.path import exists, join, basename
from tqdm import tqdm
from json import load, dump
from matplotlib import pyplot as plt
from collections import Counter

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from quadtreed3 import Quadtree, Node
from scipy.sparse import csr_matrix
from sklearn.neighbors import KernelDensity
from scipy.stats import norm
from typing import Tuple
from io import BytesIO
from umap import UMAP

import pandas as pd
import numpy as np
import ndjson
import requests
import urllib
import wizmap

SEED = 20230501

plt.rcParams["figure.dpi"] = 300
```

--------------------------------

### Extract Topics for All Leaf Nodes at All Levels of the Quadtree

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Iterates through quadtree levels to extract topics for leaf nodes. It merges leaves before a specified level, creates a tile matrix, transforms a count matrix, and then uses `get_tile_topics` to compute keywords for each level. The function returns a dictionary mapping levels to their extracted topics.

```python
def extract_level_topics(
    root: Node,
    count_mat: csr_matrix,
    texts: list[str],
    ngrams: list[str],
    min_level = None,
    max_level = None
):
    """Extract topics for all leaf nodes at all levels of the quadtree.

    Args:
        root (Noe): Quadtree node
        count_mat (csr_matrix): Count vector for the corpus
        texts (list[str]): A list of all the embeddings' texts
        ngrams (list[str]): n-gram list for the count vectorizer
    """

    level_tile_topics = {}
    
    if min_level is None:
        min_level = 0

    if max_level is None:
        max_level = root.height

    for level in tqdm(list(range(max_level, min_level - 1, -1))):
        # Create a sparse matrix
        csr_row_indexes, csr_column_indexes, row_node_map = merge_leaves_before_level(
            root, level
        )

        csr_data = [1 for _ in range(len(csr_row_indexes))]
        tile_mat = csr_matrix(
            (csr_data, (csr_row_indexes, csr_column_indexes)),
            shape=(len(texts), len(texts)),
        )

        # Transform the count matrix
        new_count_mat = tile_mat @ count_mat

        # Compute t-tf-idf scores and extract keywords
        tile_topics = get_tile_topics(new_count_mat, row_node_map, ngrams)
        
        level_tile_topics[level] = tile_topics
        
    return level_tile_topics
```

--------------------------------

### Structure and Save Topic Data for Visualization

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This code snippet structures extracted topic data into a dictionary format suitable for JSON output. It iterates through specified quadtree levels and their associated topics, extracting relevant information like topic name, coordinates, and level. The processed data is then saved to a JSON file. Dependencies include NumPy and the `dump` function for serialization.

```python
data_dict = {
    'extent': tree.extent(),
    'data': {},
    'range': [np.min(xs), np.min(ys), np.max(xs), np.max(ys)]
}

for cur_level in range(min_level, max_level + 1):
    cur_topics = level_tile_topics[cur_level]
    data_dict['data'][cur_level] = []

    for topic in cur_topics:
        # Get the topic name
        name = '-'.join([p[0] for p in topic['w'][:4]])
        x = (topic['p'][0] + topic['p'][2]) / 2
        y = (topic['p'][1] + topic['p'][3]) / 2
        cur_data = {
            'x': round(x, 3),
            'y': round(y, 3),
            'n': name,
            'l': cur_level
        }
        data_dict['data'][cur_level].append(
            [round(x, 3), round(y, 3), name]
        )

dump(data_dict, open(join(DATA_DIR, 'topic-data.json'), 'w'))
```

--------------------------------

### Prepare Data for UMAP DataFrame

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This snippet prepares lists of authors, titles, and years from a metadata dataframe. It converts authors and titles to lowercase and years to integers, likely for consistent processing in downstream tasks like UMAP projection.

```python
authors = list(map(lambda x: x.lower(), metadata_df['author']))
titles = list(map(lambda x: x.lower(), metadata_df['title']))
years = list(map(lambda x: int(x), metadata_df['year']))
```

--------------------------------

### Extract Topics for Specific Quadtree Levels

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This function extracts topics from a quadtree structure for a given range of levels. It requires the root of the quadtree, count matrix, text data, ngrams, and the min/max levels. The output is a dictionary where keys are levels and values are lists of topics. Progress is indicated by a progress bar.

```python
level_tile_topics = extract_level_topics(
    root, count_mat, texts, ngrams, min_level=min_level, max_level=max_level
)
```

--------------------------------

### Select Topic Levels for Visualization

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Automatically determines the minimum and maximum topic levels required for visualization based on SVG dimensions, data extent, and an ideal tile width. It calculates the necessary scale and iterates through potential levels to find the best fit, returning the min and max levels.

```python
def select_topic_levels(
    max_zoom_scale,
    svg_width,
    svg_height,
    x_domain,
    y_domain,
    tree_extent,
    ideal_tile_width=35,
):
    """
    Automatically determine the min and max topic levels needed for the visualization.

    Args:
        max_zoom_scale (float): Max zoom scale level
        svg_width (int): SVG width
        svg_height (int): SVG height
        x_domain ([float, float]): [x min, x max]
        y_domain ([float, float]): [y min, y max]
        tree_extent ([[float, float], [float, float]]): The extent of the tree
        ideal_tile_width (int, optional): Optimal tile width in pixel. Defaults to 35.
    """

    svg_length = max(svg_width, svg_height)
    world_length = max(x_domain[1] - x_domain[0], y_domain[1] - y_domain[0])
    tree_to_world_scale = (tree_extent[1][0] - tree_extent[0][0]) / world_length

    scale = 1
    selected_levels = []

    while scale <= max_zoom_scale:
        best_level = 1
        best_tile_width_diff = np.Infinity

        for l in range(1, 21):
            tile_num = 2**l
            svg_scaled_length = scale * svg_length * tree_to_world_scale
            tile_width = svg_scaled_length / tile_num

            if abs(tile_width - ideal_tile_width) < best_tile_width_diff:
                best_tile_width_diff = abs(tile_width - ideal_tile_width)
                best_level = l

        selected_levels.append(best_level)
        scale += 0.5

    return np.min(selected_levels), np.max(selected_levels)
```

--------------------------------

### Generate JSON Data for WizMap

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

This Python snippet generates a list of JSON strings from image names and prompts, creating data suitable for WizMap visualization. It constructs JSON objects with keys for thumbnail, tooltip, large thumbnail, and a GitHub link.

```python
json_strings = []
for i, name in enumerate(image_names):
    cur_url_part = f"{name[:2]}/{name}"
    json_data = {
        # thumbnail image name
        "i": cur_url_part,
        # tooltip text
        "t": prompts[i],
        # large thumbnail image name (optional, can be different from thumbnail)
        "li": cur_url_part,
        # link content (optional, these will be shown in the info window when clicked)
        # It can be useful to allow users to open external sources to view more information
        "github": f"https://github.com/poloclub/diffusiondb-thumbnails/blob/master/{cur_url_part}",
    }
    json_strings.append(json.dumps(json_data))
```

--------------------------------

### Save Embeddings to File - Python

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Saves the generated text embeddings to a pickle file. This allows for later retrieval without needing to recompute them.

```python
pickle.dump(embeddings, open(join(DATA_DIR, 'embeddings.pkl'), 'wb'))
```

--------------------------------

### Generate Text Embeddings - Python

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Generates numerical embeddings for a list of abstracts using a pre-initialized Sentence Transformer model. It processes abstracts in batches for efficiency and displays a progress bar.

```python
abstracts = []
for sentence in metadata_df['abstract']:
    abstracts.append(sentence.lower())

embeddings = model.encode(abstracts, batch_size=64, show_progress_bar=True)
```

--------------------------------

### Specify ACL Parquet File Path - Python

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Constructs the full path to the ACL metadata file, expected to be in Parquet format. This path is used for loading the dataset.

```python
PARQUET_PATH = join(DATA_DIR, 'acl.parquet')
```

--------------------------------

### Load UMAP CSV and Prepare for Text Extraction

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This snippet reloads the 'umap.csv' file into a DataFrame and initializes empty lists for data and texts. It then extracts the 'abstracts' column into the 'texts' list and assigns 'xs' and 'ys' columns to NumPy arrays, preparing for further text-based analysis.

```python
umap_df = pd.read_csv(join(DATA_DIR, 'umap.csv'))

data = []
texts = []

text_key = 'abstracts'
texts = umap_df[text_key]

xs = umap_df['xs']
ys = umap_df['ys']

```

--------------------------------

### Load Pre-extracted IMDB Review Embeddings

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

Loads pre-computed sentence embeddings for IMDB reviews from a specified URL. This step bypasses the time-consuming process of generating embeddings from scratch, making the notebook faster to run.

```python
# To save time, we will just load pre-extracted embeddings
EMBEDDING_URL = "https://huggingface.co/datasets/xiaohk/embeddings/resolve/main/imdb/imdb-train-25k-all-MiniLM-L6-v2-embeddings.npz"
stream = requests.get(EMBEDDING_URL, stream=True)
embedding_data = np.load(BytesIO(stream.raw.read()))

imdb_reviews = embedding_data["texts"]
imdb_embeddings = embedding_data["embeddings"]

print(f"Loaded {len(imdb_reviews)} IMDB reviews")
print(f"Embedding shape: {imdb_embeddings.shape}")
```

--------------------------------

### Print Loaded Embedding Information

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

Prints the number of loaded embeddings from the 'embedding_data.npz' file. This confirms that the embeddings have been successfully loaded and ready for further processing.

```python
image_names = embedding_data["image_names"]
prompts = embedding_data["prompts"]
embeddings = embedding_data["embeddings"]

print(f"Loaded {len(embeddings)} embeddings")
```

--------------------------------

### Define WizMap JSON File URLs (Python)

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

This Python snippet defines the URLs for the `data.ndjson` and `grid.json` files. These URLs are essential for WizMap to access the data for visualization. No external libraries are required for this basic URL definition.

```python
data_url = (
    "https://huggingface.co/datasets/xiaohk/embeddings/resolve/main/imdb/data.ndjson"
)grid_url = (
    "https://huggingface.co/datasets/xiaohk/embeddings/resolve/main/imdb/grid.json"
)
```

--------------------------------

### Create Data Array from Lists

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This Python code iterates through two lists, 'xs' and 'ys', to create a list of dictionaries. Each dictionary represents a data point with 'x', 'y' coordinates and a 'pid' (point ID) corresponding to its index. This is a common pattern for preparing data for visualization or further processing.

```python
for i in range(len(xs)):
    cur_data = {
        'x': xs[i],
        'y': ys[i],
        'pid': i,
    }
    data.append(cur_data)
```

--------------------------------

### Create Contour Plot for KDE

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This code generates a contour plot of the calculated grid density. It sets the plot limits, title, and uses 'contourf' to display the density levels with a 'Blues' colormap. The levels are determined by linearly spacing values from 0 to the maximum density.

```python
fig = plt.figure()
ax = fig.gca()

ax.set_xlim(x_min, x_max)
ax.set_ylim(y_min, y_max)

# Contourf plot
ax.set_title(f'KDE on {grid_density.shape[0]} Grid of {sample_size} Samples (bw={bw:.2f})')
cfset = ax.contourf(xx, yy, grid_density.round(4),
                    levels=np.linspace(0, np.max(grid_density), 20),
                    cmap='Blues',
                    alpha=1)
```

--------------------------------

### Extract and Save OpenAI Embeddings

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

This code snippet demonstrates how to extract embeddings for image prompts using OpenAI's text-embedding-3-large model. It then saves these embeddings along with image names and prompts to a .npz file. This process requires an OpenAI API key and the 'part-000001.json' file containing image data.

```python
# image_data = json.load(open("part-000001.json", "r"))
```

```python
# image_prompts = [(k, v["p"]) for k, v in image_data.items()]
# prompts = [item[1] for item in image_prompts]
# image_names = [item[0] for item in image_prompts]

# openai_client = AsyncOpenAI(api_key=input("OpenAI API Key: "))

# embedding_model = "text-embedding-3-large"
# results = await openai_client.embeddings.create(input=prompts, model=embedding_model)

# embeddings = [e.embedding for e in results.data]

# np.savez(
#     "embedding_data.npz",
#     embeddings=embeddings,
#     image_names=image_names,
#     prompts=prompts,
# )
```

--------------------------------

### Configure WizMap Content

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

Configures the content display settings for WizMap, specifying keys for text, images, and link fields. This allows customization of how data points are represented in the visualization.

```python
json_point_content_config = wizmap.JsonPointContentConfig(
    groupLabels=None,
    textKey="t",
    imageKey="i",
    imageURLPrefix="https://raw.githubusercontent.com/poloclub/diffusiondb-thumbnails/master/",
    largeImageKey="li",
    largeImageURLPrefix="https://raw.githubusercontent.com/poloclub/diffusiondb-thumbnails/master/",
    linkFieldKeys=["github"],
)
```

--------------------------------

### WizMap Citation in BibTeX Format

Source: https://github.com/poloclub/wizmap/blob/main/README.md

The BibTeX entry for citing the WizMap research paper. This provides all necessary details for academic referencing, including authors, title, year, and publication venue.

```bibtex
@article{wangWizMapScalableInteractive2023,
  title = {{{WizMap}}}: {{Scalable Interactive Visualization}} for {{Exploring Large Machine Learning Embeddings}},
  shorttitle = {{{WizMap}}},
  author = {Wang, Zijie J. and Hohman, Fred and Chau, Duen Horng},
  year = {2023},
  url = {http://arxiv.org/abs/2306.09328},
  urldate = {2023-06-16},
  archiveprefix = {arxiv},
  journal = {arXiv 2306.09328}
}
```

--------------------------------

### Extract Topics from Count Matrix using TF-IDF

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Computes TF-IDF scores on a count matrix to identify the top-k keywords for each tile in a quadtree. It uses the TfidfTransformer and custom sparse matrix functions. The output is a list of dictionaries, where each dictionary contains keywords ('w') and their scores, along with the tile's position ('p').

```python
def get_tile_topics(count_mat, row_pos_map, ngrams, top_k=10):
    """Get the top-k important keywords from all rows in the count_mat.

    Args:
        count_mat (csr_mat): A count matrix
        row_pos_map (dict): A dictionary that maps row index to the corresponding
            leaf node's location in the quadtree
        ngrams (list[str]): Feature names in the count_mat
        top_k (int): Number of keywords to extract
    """
    
    
    # Compute tf-idf score
    t_tf_idf_model = TfidfTransformer()
    t_tf_idf = t_tf_idf_model.fit_transform(count_mat)
    
    # Get words with top scores for each tile
    indices = top_n_idx_sparse(t_tf_idf, top_k)
    scores = top_n_values_sparse(t_tf_idf, indices)

    sorted_indices = np.argsort(scores, 1)
    indices = np.take_along_axis(indices, sorted_indices, axis=1)
    scores = np.take_along_axis(scores, sorted_indices, axis=1)
    
    # Store these keywords
    tile_topics = []

    for r in row_pos_map:
        word_scores = [
            (ngrams[word_index], round(score, 4))
            if word_index is not None and score > 0
            else ("", 0.00001)
            for word_index, score in zip(indices[r][::-1], scores[r][::-1])
        ]

        tile_topics.append({
            'w': word_scores,
            'p': row_pos_map[r]
        })

    return tile_topics
```

--------------------------------

### Determine Quadtree Levels for Data Extraction

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This function determines the minimum and maximum quadtree levels to extract based on the provided visualization parameters and the tree's extent. It takes several arguments related to SVG dimensions, coordinate domains, and the quadtree structure. The output is a tuple containing the min and max levels.

```python
min_level, max_level = select_topic_levels(
    max_zoom_scale, svg_width, svg_height, x_domain, y_domain, tree.extent()
)
print(min_level, max_level)
```

--------------------------------

### Merge Tree Leaves and Extract Data (Python)

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This Python function merges nodes in a tree structure up to a specified target level and extracts data from leaf nodes at or before that level. It constructs a sparse matrix representation where rows correspond to aggregated leaf nodes and columns represent prompt IDs. The function modifies the tree in-place and returns row indices, column indices, and a mapping from row index to leaf node.

```python
from typing import Tuple, list, dict
# Assuming Node class is defined elsewhere
# from .node import Node # Example import

def merge_leaves_before_level(root: Node, target_level: int) -> Tuple[list, list, dict]:
    """
    Merge all nodes to their parents until the tree is target_level tall (modify
    root in-place) and extract all data from leaf nodes before or at the target_level.

    Args:
        root (Node): Root node
        target_level (int): Target level

    Returns:
        csr_row_indexes (list): Row indexes for the sparse matrix. Each row is
            a leaf node.
        csr_column_indexes (list): Column indexes for the sparse matrix. Each column
            is a prompt ID.
        row_node_map (dict): A dictionary map row index to the leaf node.
    """
    
    x0, y0, x1, y1 = root.position
    step_size = (x1 - x0) / (2 ** target_level)

    # Find all leaves at or before the target level
    row_pos_map = {}
    stack = [root]

    # We create a sparse matrix by (data, (row index, column index))
    csr_row_indexes, csr_column_indexes = [], []

    # In the multiplication sparse matrix, each row represents a tile / collection,
    # and each column represents a prompt ID
    cur_r = 0

    while len(stack) > 0:
        cur_node = stack.pop()
        
        if cur_node.level >= target_level:
            # A new traverse here to concatenate all the prompts from its subtree,
            # and to merge it with its children
            local_stack = [cur_node]
            subtree_data = []

            while len(local_stack) > 0:
                local_node = local_stack.pop()
                
                if len(local_node.children) == 0:
                    # Leaf node
                    subtree_data.extend(local_node.data)
                else:
                    for c in local_node.children[::-1]:
                        if c is not None:
                            local_stack.append(c)
                            
            # Detach all the children and get their data
            cur_node.children = []
            cur_node.data = subtree_data
            
            # Register this node in a dictionary for faster access
            row_pos_map[cur_r] =  list(map(lambda x: round(x, 3), cur_node.position))
            
            # Collect the prompt IDs
            for d in cur_node.data:
                csr_row_indexes.append(cur_r)
                csr_column_indexes.append(d['pid'])
                
            # Move on to the next tile / collection
            cur_r += 1

        else:
            if len(cur_node.children) == 0:
                # Leaf node => it means this leaf is before the target level
                # We need to adjust the node's position so that it has the same
                # size as leaf nodes at the target_level
                x, y = cur_node.data[0]['x'], cur_node.data[0]['y']
                xi, yi = int((x - x0) // step_size), int((y - y0) // step_size)

                # Find the bounding box of current level of this leaf node
                xi0, yi0 = x0 + xi * step_size, y0 + yi * step_size
                xi1, yi1 = xi0 + step_size, yi0 + step_size
                row_pos_map[cur_r] = list(map(lambda x: round(x, 3), [xi0, yi0, xi1, yi1]))
                
                # Collect the prompt IDs
                for d in cur_node.data:
                    csr_row_indexes.append(cur_r)
                    csr_column_indexes.append(d['pid'])
                
                # Move on to the next tile / collection
                cur_r += 1
            else:
                for c in cur_node.children[::-1]:
                    if c is not None:
                        stack.append(c)

    return csr_row_indexes, csr_column_indexes, row_pos_map
```

--------------------------------

### Save WizMap JSON Files

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

Saves the generated `data_list` and `grid_dict` into two separate JSON files in the specified output directory. These files are the final output required for WizMap to visualize the dataset.

```python
# Save the JSON files
wizmap.save_json_files(data_list, grid_dict, output_dir="./")
```

--------------------------------

### Prepare Data for WizMap JSON Generation

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

Extracts the 2D coordinates (x, y) and the original IMDB review texts from the processed data. This data will be used as input for WizMap's JSON generation functions.

```python
xs = embeddings_2d[:, 0].astype(float).tolist()
ys = embeddings_2d[:, 1].astype(float).tolist()
texts = imdb_reviews
```

--------------------------------

### Define WizMap Data URLs

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

Defines the URLs for the generated data.ndjson and grid.json files. These URLs are used to load data into the WizMap visualization.

```python
data_url = "https://huggingface.co/datasets/xiaohk/embeddings/resolve/main/diffusiondb/data.ndjson"
grid_url = "https://huggingface.co/datasets/xiaohk/embeddings/resolve/main/diffusiondb/grid.json"
```

--------------------------------

### Prepare and Save UMAP Data in NDJSON Format

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This Python script reads UMAP data from a CSV file, processes it based on specified text, time, and label keys, and then saves the formatted data into an NDJSON file. It handles optional time and label keys and constructs rows containing coordinates, text, and potentially time and labels. Dependencies include Pandas and the `ndjson` library.

```python
umap_df = pd.read_csv(join(DATA_DIR, 'umap.csv'))

text_key = 'abstracts'
time_key = 'years'
label_key = 'titles'

texts = umap_df[text_key]

if time_key is not None:
    times = list(map(str, umap_df[time_key]))
else:
    times = []
    
if label_key is not None:
    labels = umap_df[label_key]
else:
    labels = []

xs = umap_df['xs']
ys = umap_df['ys']

umap_data_short = []

for i in range(len(xs)):
    cur_row = [xs[i], ys[i], texts[i]]
    
    if time_key is not None:
        cur_row.append(times[i])
        
        if label_key is not None:
            cur_row.append(labels[i])
    
    else:
        if label_key is not None:
            cur_row.append('')
            cur_row.append(labels[i])

    umap_data_short.append(cur_row)
    
with open(join(DATA_DIR, "./umap.ndjson"), 'w') as fp:
    ndjson.dump(umap_data_short, fp)
```

--------------------------------

### Visualize UMAP Projection - Python

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

Creates a scatter plot of the 2D projected embeddings generated by UMAP. The plot is titled with the UMAP parameters used and displays the points with small size and low opacity for better visualization of dense data.

```python
plt.title(f'n_neighbors={n_neighbors}, min_dist={min_dist}')
plt.scatter(projected_emb_cuml[:, 0], projected_emb_cuml[:, 1], s=0.1, alpha=0.2)
```

--------------------------------

### Create UMAP DataFrame with Projected Embeddings

Source: https://github.com/poloclub/wizmap/blob/main/example/acl-abstracts.ipynb

This code constructs a Pandas DataFrame named 'umap_df'. It combines projected embeddings (xs, ys) with abstracts, titles, years, and authors, creating a consolidated dataset for UMAP analysis.

```python
umap_df = pd.DataFrame({
    'xs': projected_emb_cuml[:, 0],
    'ys': projected_emb_cuml[:, 1],
    'abstracts': abstracts,
    'titles': titles,
    'years': years,
    'authors': authors
})
```

--------------------------------

### Visualize 2D Projected Embeddings

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

Generates a scatter plot of the 2D projected embeddings using Matplotlib. This visualization provides an initial look at the structure of the data in the reduced dimensional space.

```python
plt.title(f"UMAP Projected Embeddings of {len(imdb_reviews)} IMDB Reviews")
plt.scatter(embeddings_2d[:, 0], embeddings_2d[:, 1], s=0.1, alpha=0.2)
plt.show()
```

--------------------------------

### Generate WizMap Grid Data

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

Generates a dictionary representing the grid data for WizMap. This function takes x and y coordinates, text data, and optional configuration for content display.

```python
grid_data = wizmap.generate_grid_dict(
    xs=xs,
    ys=ys,
    texts=json_strings,
)

data_list = wizmap.generate_data_list(xs, ys, json_strings)
grid_dict = wizmap.generate_grid_dict(
    xs, # This is a duplicate from the previous call, likely a typo in the original text
    ys,
    json_strings,
    "DiffusionDB Images",
    json_point_content_config=json_point_content_config,
)
```

--------------------------------

### Generate WizMap JSON Data Structures

Source: https://github.com/poloclub/wizmap/blob/main/example/imdb.ipynb

Utilizes the WizMap library to generate two essential data structures: `data_list` for raw data encoding and `grid_dict` for contour plot and multi-level summaries. These are prerequisites for creating the final WizMap JSON files.

```python
data_list = wizmap.generate_data_list(xs, ys, texts)
grid_dict = wizmap.generate_grid_dict(xs, ys, texts, "IMDB Reviews")
```

--------------------------------

### Prepare Embedding Coordinates for WizMap

Source: https://github.com/poloclub/wizmap/blob/main/example/diffusiondb-images.ipynb

Extracts and rounds the 2D embedding coordinates to 5 decimal places. These rounded coordinates are crucial for generating the JSON files required by WizMap.

```python
# Extract and round the 2D embedding coordinates to 5 decimal places
xs = [round(float(x), 5) for x in embeddings_2d[:, 0]]
ys = [round(float(y), 5) for y in embeddings_2d[:, 1]]
```