### Command Line Usage for PyAtlas Setup and Development Source: https://context7.com/fpgmaas/pyatlas/llms.txt Provides essential command-line instructions for setting up and running the PyAtlas project. Includes commands for installing dependencies, running the full setup pipeline, executing individual scripts, running tests, and starting the frontend development server. ```bash # Install dependencies make install # Run the complete setup pipeline uv run python pyatlas/scripts/setup.py # Run individual scripts uv run python pyatlas/scripts/download_raw_dataset.py uv run python pyatlas/scripts/process_raw_dataset.py uv run python pyatlas/scripts/create_vector_embeddings.py uv run python pyatlas/scripts/generate_clusters.py uv run python pyatlas/scripts/generate_cluster_labels.py uv run python pyatlas/scripts/calculate_cluster_metadata.py uv run python pyatlas/scripts/generate_json_outputs.py # Run tests make test # Start frontend development server cd frontend && npm install && npm run dev # Access at http://localhost:5173 # Start frontend with remote data (no local processing needed) cd frontend && npm run dev:remote ``` -------------------------------- ### Python Script for Complete PyAtlas Setup Pipeline Source: https://context7.com/fpgmaas/pyatlas/llms.txt This Python script, `main.py`, orchestrates the entire data processing pipeline for PyAtlas. It handles downloading data, processing it, creating embeddings, generating clusters, labeling clusters, calculating metadata, and finally generating JSON outputs. ```python from pyatlas.scripts.setup import main # Run complete pipeline: # 1. Download raw dataset from Google Drive # 2. Process raw dataset (filter, clean descriptions) # 3. Create vector embeddings using Sentence Transformers # 4. Generate clusters using UMAP + HDBSCAN # 5. Generate cluster labels using OpenAI # 6. Calculate cluster metadata (centroids, bounds) # 7. Generate JSON outputs for frontend main() ``` -------------------------------- ### Environment Variables for PyAtlas Configuration Source: https://context7.com/fpgmaas/pyatlas/llms.txt Shows examples of environment variables used to configure the PyAtlas application. These variables allow for overriding default settings for OpenAI API key, embedding models, data folders, and ETL parameters. ```bash # .env file configuration PYATLAS__openai_api_key=sk-your-api-key # Optional overrides PYATLAS_APP__embeddings_model_name=all-MiniLM-L6-v2 PYATLAS_STORAGE__data_folder=custom_data PYATLAS_ETL__top_packages_to_include=5000 PYATLAS_ETL__overwrite_download=true ``` -------------------------------- ### Python Configuration Loading with PyAtlas Source: https://context7.com/fpgmaas/pyatlas/llms.txt Demonstrates how to load and access application configuration using the PyAtlas Config class. Configuration can be managed via environment variables or a .env file, providing access to app, storage, ETL, and OpenAI settings. ```python from pyatlas.config import Config, AppConfig, StorageConfig, ETLConfig, OpenAIConfig # Load configuration config = Config.load() # Access configuration sections print(config.app.name) # "pyatlas" print(config.app.embeddings_model_name) # "all-mpnet-base-v2" print(config.storage.data_folder) # Path("data") print(config.storage.packages_json) # "packages.json" print(config.etl.top_packages_to_include) # 10000 print(config.etl.google_file_id) # Google Drive file ID print(config.openai.model_name) # "gpt-5-mini" # Dump all config for debugging print(config.dump()) ``` -------------------------------- ### Manage Galaxy Store State with Zustand (TypeScript) Source: https://context7.com/fpgmaas/pyatlas/llms.txt Demonstrates how to use Zustand for state management in a React component for the Galaxy Store visualization. It covers selecting packages, focusing on clusters, and handling search queries. Dependencies include React and Zustand. ```typescript import { useGalaxyStore } from "./store/useGalaxyStore"; // In a React component function MyComponent() { const { packages, clusters, selectedPackageId, setSelectedPackageId, focusOnCluster, searchQuery, setSearchQuery, } = useGalaxyStore(); // Select a package const handlePackageClick = (packageId: number) => { setSelectedPackageId(packageId); }; // Focus camera on a cluster const handleClusterClick = (clusterId: number) => { focusOnCluster(clusterId); // Animates camera to cluster centroid and highlights cluster }; // Search packages const handleSearch = (query: string) => { setSearchQuery(query); }; return (/* ... */); } ``` -------------------------------- ### Download Raw PyPI Dataset using Python Source: https://context7.com/fpgmaas/pyatlas/llms.txt Downloads the raw PyPI dataset from Google Drive. Configuration for the Google Drive file ID and overwrite behavior is managed via environment variables. ```python from pyatlas.scripts.download_raw_dataset import download_raw_dataset from pyatlas.config import Config # Configuration is loaded from .env file # PYATLAS_ETL__google_file_id - Google Drive file ID # PYATLAS_ETL__overwrite_download - Whether to overwrite existing files download_raw_dataset() # Output: Downloads raw_dataset.csv to data/ folder # Logs: "Downloading raw dataset from Google Drive to data/raw_dataset.csv..." ``` -------------------------------- ### TypeScript Data Loader Functions for React Frontend Source: https://context7.com/fpgmaas/pyatlas/llms.txt Illustrates TypeScript functions used in the React frontend to asynchronously load package, cluster, and constellation data. It includes type definitions for Package and Cluster interfaces and demonstrates how to handle potential loading errors. ```typescript import { loadPackages, loadClusters, loadConstellations } from "./utils/dataLoader"; import type { Package, Cluster, Constellation } from "./types"; // Load all data async function loadData() { try { const [packages, clusters, constellations] = await Promise.all([ loadPackages(), loadClusters(), loadConstellations(), ]); console.log(`Loaded ${packages.length} packages`); console.log(`Loaded ${clusters.length} clusters`); console.log(`Loaded ${constellations.length} constellation edges`); return { packages, clusters, constellations }; } catch (error) { console.error("Failed to load data:", error); throw error; } } // Package interface interface Package { id: number; name: string; summary: string; downloads: number; x: number; y: number; clusterId: number; } // Cluster interface interface Cluster { clusterId: number; label: string; centroidX: number; centroidY: number; downloads: number; minX: number; maxX: number; minY: number; maxY: number; } ``` -------------------------------- ### Render and Extract Text from Descriptions using Python Source: https://context7.com/fpgmaas/pyatlas/llms.txt Renders package descriptions from different formats (Markdown, RST, plain text) into plain text suitable for embedding generation. It handles the conversion and passthrough of various content types. ```python from pyatlas.data.description.description_parser import render_and_extract_text from pyatlas.data.description.content_type_detector import ContentType # Convert Markdown to plain text markdown = "# Hello\n\nThis is **bold** and [a link](https://example.com)." text = render_and_extract_text(markdown, ContentType.MARKDOWN) # Output: "Hello\nThis is bold and a link." # Convert RST to plain text rst = ".. note::\n\n Important information here." text = render_and_extract_text(rst, ContentType.RST) # Output: Extracted plain text from RST # Plain text passthrough plain = "Simple description without formatting." text = render_and_extract_text(plain, ContentType.PLAIN) # Output: "Simple description without formatting." ``` -------------------------------- ### Query Top PyPI Packages from BigQuery (SQL) Source: https://context7.com/fpgmaas/pyatlas/llms.txt A SQL query designed for Google BigQuery to extract the top Python packages based on weekly downloads from the public PyPI dataset. It joins download data with distribution metadata to provide package names, descriptions, and download counts. Requires access to Google BigQuery. ```sql -- pypi_bigquery.sql -- Extracts top packages by weekly downloads with metadata WITH recent_downloads AS ( SELECT LOWER(project) AS project_lower, project, COUNT(*) AS download_count FROM `bigquery-public-data.pypi.file_downloads` WHERE DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) AND CURRENT_DATE() GROUP BY LOWER(project), project HAVING COUNT(*) >= 100 ), latest_metadata AS ( SELECT LOWER(name) AS name_lower, name, description, summary, version, upload_time, ROW_NUMBER() OVER (PARTITION BY LOWER(name) ORDER BY upload_time DESC) AS rn FROM `bigquery-public-data.pypi.distribution_metadata` ) SELECT lm.name AS name, lm.description AS description, description_content_type, lm.summary AS summary, lm.version AS latest_version, rd.download_count AS number_of_downloads FROM recent_downloads rd JOIN latest_metadata lm ON rd.project_lower = lm.name_lower WHERE lm.rn = 1 ORDER BY rd.download_count DESC; ``` -------------------------------- ### Detect Content Type of Package Descriptions using Python Source: https://context7.com/fpgmaas/pyatlas/llms.txt Detects the content type (Markdown, RST, or plain text) of package descriptions. It uses provided metadata or heuristic pattern matching. Short or empty text defaults to PLAIN. ```python from pyatlas.data.description.content_type_detector import ContentTypeDetector, ContentType detector = ContentTypeDetector() # Detection from metadata result = detector.detect("any text", content_type="text/markdown") assert result == ContentType.MARKDOWN result = detector.detect("any text", content_type="text/x-rst") assert result == ContentType.RST # Heuristic detection (when no metadata available) markdown_text = "# Header\n\nCheck [this link](https://example.com) for info." result = detector.detect(markdown_text) assert result == ContentType.MARKDOWN rst_text = ".. image:: logo.png\n\n.. note::\n\n Important info." result = detector.detect(rst_text) assert result == ContentType.RST # Short or empty text defaults to PLAIN result = detector.detect("short") assert result == ContentType.PLAIN ``` -------------------------------- ### Process Raw PyPI Dataset using Python Source: https://context7.com/fpgmaas/pyatlas/llms.txt Processes raw PyPI data by filtering for the top N packages and cleaning the text from package descriptions. It handles various text formats like Markdown and RST. The full pipeline can be run to create a processed dataset file. ```python from pyatlas.scripts.process_raw_dataset import process_raw_dataset from pyatlas.data.raw_data_reader import RawDataReader from pyatlas.data.description.description_cleaner import DescriptionCleaner import polars as pl # Read and process raw data reader = RawDataReader(raw_dataset="data/raw_dataset.csv") df = reader.read() # Returns DataFrame with columns: name, summary, description, weekly_downloads # Keep only top N packages df = df.sort("weekly_downloads", descending=True).head(10000) # Extract plain text from descriptions (handles Markdown, RST, plain text) cleaner = DescriptionCleaner() df = cleaner.extract_text(df, input_col="description", output_col="description_cleaned") # Output: Adds 'description_cleaned' column with extracted text # Or run the full pipeline process_raw_dataset() # Output: Creates data/processed_dataset.csv ``` -------------------------------- ### Generate Package Constellations (Python) Source: https://context7.com/fpgmaas/pyatlas/llms.txt The ClusterConstellationsGenerator creates constellation edges connecting top packages within clusters using minimum spanning trees. It allows configuration of parameters like the percentage of top packages to consider, minimum packages per cluster, and maximum edge length fraction. The output is a DataFrame detailing the connections between packages. ```python from pyatlas.clustering.constellations import ClusterConstellationsGenerator import polars as pl # Sample data with coordinates and downloads df = pl.DataFrame({ "name": ["a", "b", "c", "d", "e", "f"], "x": [0.0, 0.1, 0.05, 0.5, 0.6, 0.55], "y": [0.0, 0.0, 0.1, 0.5, 0.5, 0.6], "cluster_id": ["1", "1", "1", "2", "2", "2"], "weekly_downloads": [100, 200, 150, 300, 250, 350] }) # Generate constellation edges generator = ClusterConstellationsGenerator( top_percent=0.5, # Use top 50% of packages by downloads min_packages=3, # Minimum packages required for constellation cutoff_length_frac=0.05 # Maximum edge length (fraction of total range) ) constellations = generator.generate_constellations(df) # Output: DataFrame with edge coordinates print(constellations) # shape: (4, 7) # ┌────────────┬──────────────┬────────────┬────────┬────────┬───────┬───────┐ # │ cluster_id ┆ from_package ┆ to_package ┆ from_x ┆ from_y ┆ to_x ┆ to_y │ # │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ # │ str ┆ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ # ╞════════════╪══════════════╪════════════╪════════╪════════╪═══════╪═══════╡ # │ 1 ┆ b ┆ a ┆ 0.1 ┆ 0.0 ┆ 0.0 ┆ 0.0 │ # │ 1 ┆ c ┆ a ┆ 0.05 ┆ 0.1 ┆ 0.0 ┆ 0.0 │ # │ 2 ┆ f ┆ d ┆ 0.55 ┆ 0.6 ┆ 0.5 ┆ 0.5 │ # │ 2 ┆ e ┆ d ┆ 0.6 ┆ 0.5 ┆ 0.5 ┆ 0.5 │ # └────────────┴──────────────┴────────────┴────────┴────────┴───────┴───────┘ ``` -------------------------------- ### Generate JSON Outputs Script (Python) Source: https://context7.com/fpgmaas/pyatlas/llms.txt This script orchestrates the generation of JSON output files required for frontend consumption. It reads various intermediate CSV files (clustered_dataset.csv, cluster_metadata.csv, cluster_labels.csv, constellations.csv) and consolidates the data into three main JSON files: packages.json, clusters.json, and constellations.json. ```python from pyatlas.scripts.generate_json_outputs import generate_json_outputs # Reads clustered_dataset.csv, cluster_metadata.csv, cluster_labels.csv, constellations.csv generate_json_outputs() # Output files: # data/packages.json - Package data for visualization # data/clusters.json - Cluster metadata and labels # data/constellations.json - Constellation edge data ``` -------------------------------- ### Create Sample DataFrame and Add Embeddings (Polars) Source: https://context7.com/fpgmaas/pyatlas/llms.txt Demonstrates how to create a Polars DataFrame and add an 'embeddings' column using a 'creator' object. The embeddings are 768-dimensional vectors. ```python import polars as pl # Create sample DataFrame df = pl.DataFrame({ "name": ["numpy", "pandas", "scikit-learn"], "text": [ "NumPy is the fundamental package for scientific computing", "Pandas provides data structures for data analysis", "Scikit-learn is a machine learning library" ] }) # Add embeddings column # Assuming 'creator' is an initialized object with an add_embeddings method # df_with_embeddings = creator.add_embeddings(df, text_column="text") # Access embeddings # embeddings = df_with_embeddings["embeddings"].to_list() # print(f"Embedding shape: {len(embeddings[0])} dimensions") ``` -------------------------------- ### Generate Cluster Labels Script (Python) Source: https://context7.com/fpgmaas/pyatlas/llms.txt This script reads a clustered dataset (clustered_dataset.csv) and utilizes OpenAI to generate descriptive labels for each cluster. The output is saved to data/cluster_labels.csv, containing 'cluster_id' and 'cluster_label' columns. ```python from pyatlas.scripts.generate_cluster_labels import generate_cluster_labels # Reads clustered_dataset.csv, generates labels via OpenAI generate_cluster_labels() # Output: data/cluster_labels.csv with columns: cluster_id, cluster_label ``` -------------------------------- ### Generate Clusters Script (Python) Source: https://context7.com/fpgmaas/pyatlas/llms.txt A complete pipeline script to cluster packages and generate 2D coordinates. It reads from 'embeddings.parquet' and 'processed_dataset.csv', then outputs 'clustered_dataset.csv'. ```python from pyatlas.scripts.generate_clusters import generate_clusters # Reads embeddings.parquet and processed_dataset.csv # Generates cluster IDs and 2D coordinates generate_clusters() # Output: data/clustered_dataset.csv with columns: # name, weekly_downloads, summary, cluster_id, x, y ``` -------------------------------- ### Label Clusters using OpenAI API (Polars) Source: https://context7.com/fpgmaas/pyatlas/llms.txt Generates descriptive labels for clusters using the OpenAI API. It takes package names, summaries, and cluster IDs as input and outputs a DataFrame with cluster labels. Requires OpenAI API key configuration. ```python from pyatlas.clustering.labeling import ClusterLabeler from pyatlas.config import Config import polars as pl # Sample clustered data df = pl.DataFrame({ "name": ["numpy", "scipy", "pandas", "matplotlib", "seaborn"], "summary": [ "Fundamental package for scientific computing", "Scientific computing library", "Data analysis and manipulation tool", "Plotting library", "Statistical data visualization" ], "cluster_id": ["1", "1", "1", "2", "2"] }) # Generate labels (requires PYATLAS__openai_api_key in .env) labeler = ClusterLabeler( summary_column="summary", cluster_id_column="cluster_id", name_column="name", max_chars_per_summary=256, max_total_chars=16384 ) labels_df = labeler.generate_cluster_labels(df) # Output: DataFrame with cluster_id and cluster_label # shape: (2, 2) # ┌────────────┬─────────────────────────┐ # │ cluster_id ┆ cluster_label │ # │ --- ┆ --- │ ``` -------------------------------- ### Create Vector Embeddings from Text using Python Source: https://context7.com/fpgmaas/pyatlas/llms.txt Generates semantic vector embeddings from text using Sentence Transformers models. It allows initialization with a specific model and configuration of batch size and embedding column name. ```python from pyatlas.embeddings.embeddings_creator import VectorEmbeddingCreator from sentence_transformers import SentenceTransformer import polars as pl # Initialize with embedding model model = SentenceTransformer("all-mpnet-base-v2") creator = VectorEmbeddingCreator( embeddings_model=model, embedding_column_name="embeddings", batch_size=128 ) ``` -------------------------------- ### Calculate Cluster Metadata (Python) Source: https://context7.com/fpgmaas/pyatlas/llms.txt The ClusterMetadataGenerator class calculates essential metadata for each cluster, including centroids, bounding boxes, and total downloads. It takes a Polars DataFrame with cluster information as input and outputs a DataFrame containing the calculated metadata. ```python from pyatlas.clustering.metadata import ClusterMetadataGenerator import polars as pl # Sample clustered data with coordinates df = pl.DataFrame({ "name": ["a", "b", "c", "d", "e", "f"], "x": [0.0, 0.1, 0.05, 0.5, 0.6, 0.55], "y": [0.0, 0.0, 0.1, 0.5, 0.5, 0.6], "cluster_id": ["1", "1", "1", "2", "2", "2"], "weekly_downloads": [100000, 200000, 150000, 300000, 250000, 350000] }) # Generate metadata generator = ClusterMetadataGenerator() metadata = generator.generate_cluster_metadata(df) # Output: DataFrame with cluster centroids and bounds print(metadata) # shape: (2, 8) # ┌────────────┬────────────┬────────────┬─────────────────────┬───────┬───────┬───────┬───────┐ # │ cluster_id ┆ centroid_x ┆ centroid_y ┆ total_weekly_downl… ┆ min_x ┆ max_x ┆ min_y ┆ max_y │ # │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ # │ str ┆ f64 ┆ f64 ┆ i64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ # ╞════════════╪════════════╪════════════╪═════════════════════╪═══════╪═══════╪═══════╪═══════╡ # │ 1 ┆ 0.05 ┆ 0.033 ┆ 450000 ┆ 0.0 ┆ 0.1 ┆ 0.0 ┆ 0.1 │ # │ 2 ┆ 0.55 ┆ 0.533 ┆ 900000 ┆ 0.5 ┆ 0.6 ┆ 0.5 ┆ 0.6 │ # └────────────┴────────────┴────────────┴─────────────────────┴───────┴───────┴───────┴───────┘ ``` -------------------------------- ### JSON Data Structures for PyAtlas Source: https://context7.com/fpgmaas/pyatlas/llms.txt Defines the structure for packages.json, clusters.json, and constellations.json. These files contain data related to software packages, their clustering, and connections between them, used for visualization and analysis. ```json // packages.json structure [ { "id": 0, "name": "boto3", "summary": "The AWS SDK for Python", "downloads": 350000000, "x": -1.234, "y": 2.567, "clusterId": "42" } ] // clusters.json structure [ { "clusterId": "42", "label": "AWS Cloud Services", "centroidX": -1.2, "centroidY": 2.5, "downloads": 1500000000, "minX": -2.0, "maxX": -0.5, "minY": 1.5, "maxY": 3.5 } ] // constellations.json structure [ { "clusterId": "42", "fromId": 0, "toId": 15, "fromX": -1.234, "fromY": 2.567, "toX": -1.156, "toY": 2.789 } ] ``` -------------------------------- ### Generate 2D Visualization Coordinates (Polars, NumPy) Source: https://context7.com/fpgmaas/pyatlas/llms.txt Generates 2D visualization coordinates for packages using supervised UMAP with provided cluster labels. Requires NumPy for sample embedding generation and Polars for DataFrame operations. ```python from pyatlas.clustering.coordinates import ClusterCoordinatesGenerator import polars as pl import numpy as np # Sample clustered data df = pl.DataFrame({ "name": ["pkg_a", "pkg_b", "pkg_c", "pkg_d"], "embeddings": [np.random.randn(768).tolist() for _ in range(4)], "cluster_id": ["1", "1", "2", "2"] }) # Generate 2D coordinates generator = ClusterCoordinatesGenerator() df_with_coords = generator.generate_coordinates( df, embeddings_column="embeddings", cluster_id_column="cluster_id" ) # Output: DataFrame with 'x' and 'y' columns print(df_with_coords.select(["name", "x", "y"])) ``` -------------------------------- ### Generate Vector Embeddings Script (Python) Source: https://context7.com/fpgmaas/pyatlas/llms.txt A complete Python script to generate vector embeddings for processed package data. It reads from 'processed_dataset.csv' and saves the embeddings to 'embeddings.parquet'. ```python from pyatlas.scripts.create_vector_embeddings import create_vector_embeddings # Reads processed_dataset.csv, generates embeddings, saves to embeddings.parquet create_vector_embeddings() # Output: data/embeddings.parquet with columns: name, embeddings ``` -------------------------------- ### Cluster Packages using UMAP and HDBSCAN (Polars, NumPy) Source: https://context7.com/fpgmaas/pyatlas/llms.txt Assigns cluster IDs to packages using UMAP for dimensionality reduction and HDBSCAN for clustering. Handles noise points by assigning them a cluster_id of '-1'. Requires NumPy for sample data generation and Polars for DataFrame manipulation. ```python from pyatlas.clustering.clustering import ClusterIdGenerator import polars as pl import numpy as np # Create sample data with embeddings np.random.seed(42) n_samples = 100 embedding_dim = 768 # Generate embeddings with cluster structure embeddings = [] for i in range(3): # 3 clusters center = np.random.randn(embedding_dim) * 5 cluster_points = center + np.random.randn(33, embedding_dim) * 0.5 embeddings.extend(cluster_points.tolist()) embeddings.append((np.random.randn(embedding_dim) * 5).tolist()) # Extra point df = pl.DataFrame({ "name": [f"package_{i}" for i in range(n_samples)], "embeddings": embeddings }) # Generate cluster IDs generator = ClusterIdGenerator( min_cluster_size=8, min_samples=2, cluster_selection_method="leaf", cluster_selection_epsilon=0.0085 ) df_clustered = generator.generate_cluster_ids(df, embeddings_column="embeddings") # Output: DataFrame with 'cluster_id' column (string type) # Noise points get cluster_id = "-1" print(f"Unique clusters: {df_clustered['cluster_id'].n_unique()}") print(f"Noise points: {len(df_clustered.filter(pl.col('cluster_id') == '-1'))}") ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.