### Quick Start Code Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt A minimal code example to get started quickly with Autodistill Grounded SAM 2. ```python from autodistill_grounded_sam_2 import GroundedSAM2 model = GroundedSAM2(ontology=["object"]) # Load image and predict... ``` -------------------------------- ### Install autodistill-grounded-sam-2 from Source Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/08-module-exports.md Clone the repository and install the package in editable mode for development. ```bash git clone https://github.com/autodistill/autodistill-grounded-sam-2.git cd autodistill-grounded-sam-2 pip install -e . ``` -------------------------------- ### Development Installation Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Clone the repository and install in editable mode with development dependencies for local development. ```bash # Clone repository git clone https://github.com/autodistill/autodistill-grounded-sam-2.git cd autodistill-grounded-sam-2 # Install in editable mode with dev dependencies pip install -e ".[dev]" ``` -------------------------------- ### Basic Usage Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt A simple example demonstrating how to initialize the GroundedSAM2 model and use its predict method. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from supervision import Detections import cv2 # Load the model model = GroundedSAM2( ontology=["person", "bicycle"], device="cuda:0" ) # Load an image image = cv2.imread("path/to/your/image.jpg") # Perform prediction results: Detections = model.predict(image=image) # Process results (e.g., draw bounding boxes) print(results) ``` -------------------------------- ### Configuration and Initialization Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/README.md Guide to constructor options, parameters, model selection, and environment variables for configuration. ```APIDOC ## Configuration and Initialization ### Description Explains how to configure and initialize the package, covering constructor options, model selection, and environment variables. ### Constructor Options - Details the available options and parameters for the constructor. ### Model Selection - Information on choosing between different models (e.g., Florence 2 vs. Grounding DINO). ### Threshold Tuning - Guidance on adjusting thresholds for optimal performance. ### Ontology Configuration - How to configure the caption ontology. ### Environment Variables - Lists and describes relevant environment variables. ### Device and Memory Management - Options for managing device placement and memory usage. ### Model Caching Strategy - Details the strategy for caching models. ``` -------------------------------- ### Install autodistill-grounded-sam-2 from PyPI Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/08-module-exports.md Use this command to install the package from the Python Package Index. ```bash pip install autodistill-grounded-sam-2 ``` -------------------------------- ### Install PyTorch with CUDA Support Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Install PyTorch with CUDA support before installing the Autodistill package for GPU acceleration. ```bash # Install PyTorch with CUDA pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 # Then install the package pip install autodistill-grounded-sam-2 ``` -------------------------------- ### Configuration Options Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Demonstrates setting configuration options during GroundedSAM2 initialization, including ontology, device, and threshold. ```python from autodistill_grounded_sam_2 import GroundedSAM2 model = GroundedSAM2( ontology=["cat", "dog"], device="cpu", threshold=0.5 ) ``` -------------------------------- ### Install SAM 2 Locally Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Install the SAM 2 package in editable mode using pip. This is required for SAM 2 setup. ```bash pip install -e . ``` -------------------------------- ### Troubleshooting Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt A placeholder for common troubleshooting steps, often involving checking dependencies or environment variables. ```bash # Check if CUDA is available: nvidia-smi # Check installed packages: pip list | grep autodistill ``` -------------------------------- ### Configuration Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Reference for setup and configuration options, including constructor parameters, model selection, and environment variables. ```APIDOC ## Configuration ### Constructor Options Reference for all parameters available in the `GroundedSAM2` constructor, including their defaults and descriptions. ### Model Selection Guide on selecting between different models like 'Florence 2' and 'Grounding DINO'. ### Threshold Tuning Specifications for tuning confidence and IoU thresholds. ### Ontology Configuration Details on how to configure the `CaptionOntology`. ### Environment Variables List of supported environment variables for configuring the package. ### Device and Memory Management Options for selecting the device (CPU/GPU) and managing memory usage. ### Model Caching Details on how model caching is handled, including cache location (`~/.cache/autodistill/`). ### Configuration Scenarios Examples of common configuration setups. ``` -------------------------------- ### Dependencies and Requirements Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/README.md Lists runtime and development dependencies, version constraints, and installation instructions. ```APIDOC ## Dependencies and Requirements ### Description Details the runtime and development dependencies, version constraints, and installation instructions for the package. ### Runtime Dependencies - Lists all dependencies required at runtime. ### Development Dependencies - Lists dependencies needed for development and testing. ### Version Constraints - Specifies version compatibility matrix for dependencies. ### Model Download URLs - URLs for downloading required models. ### Installation Instructions - Step-by-step guide for installing the package and its dependencies. ### External Tools - Lists any required external tools. ### Network Requirements - Specifies any network-related requirements. ### Known Issues - Lists known issues and potential workarounds. ### Security Considerations - Notes on security aspects. ``` -------------------------------- ### Model Comparison Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Compare detection results from different models. This example assumes you have loaded multiple models. ```python # Assuming 'model_florence' and 'model_dino' are initialized models # and 'image' is loaded results_florence = model_florence.predict(image=image) results_dino = model_dino.predict(image=image) # Compare results... print("Florence 2 results:", results_florence) print("Grounding DINO results:", results_dino) ``` -------------------------------- ### Install Autodistill Grounded SAM 2 and YOLOv8 Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/README.md Install the necessary libraries for Autodistill Grounded SAM 2 and YOLOv8 using pip. This is the first step before using the library. ```bash pip install autodistill-grounded-sam-2 autodistill-yolov8 ``` -------------------------------- ### Usage Examples Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/README.md Illustrates basic and advanced usage patterns, model selection, threshold tuning, and working with results. ```APIDOC ## Usage Examples ### Description Demonstrates various usage patterns, from basic to advanced, including model selection, threshold tuning, and handling detection results. ### Basic Usage - Examples of fundamental usage patterns. ### Model Comparison - Examples illustrating model comparison and selection. ### Threshold Tuning Strategies - Examples showing different threshold tuning approaches. ### Working with Detection Results - How to effectively process and utilize detection results. ### Visualization Techniques - Examples of visualizing detection outputs. ### Auto-labeling Directories - Usage patterns for auto-labeling directories. ### Error Handling Patterns - Examples of implementing error handling. ### Advanced Ontologies - Examples using advanced ontology configurations. ### Performance Optimization - Techniques and examples for optimizing performance. ``` -------------------------------- ### Quick Start: Detect Objects in an Image Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Initialize the model with an ontology, predict detections on an image, and access the results. The ontology defines the objects to be detected. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology # Create ontology (what to detect) ontology = CaptionOntology({ "person": "person", "dog": "dog" }) # Create model model = GroundedSAM2(ontology=ontology) # Predict on image detections = model.predict("image.jpg") # Access results print(f"Found {len(detections)} objects") for i, box in enumerate(detections.xyxy): print(f"Box {i}: {box}") ``` -------------------------------- ### Install Autodistill Grounded SAM 2 and YOLOv8 Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/README.md Install the GroundedSAM Base Model along with a Target Model that supports the `detection` task. This command installs the necessary packages for using Grounded SAM 2 with YOLOv8. ```bash pip3 install autodistill-grounded-sam-2 autodistill-yolov8 ``` -------------------------------- ### CPU-Only Installation Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Install the CPU version of PyTorch for systems without CUDA. Note that runtime performance will be significantly slower. ```bash # Install PyTorch CPU version pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu # Install package (will be very slow at runtime) pip install autodistill-grounded-sam-2 ``` -------------------------------- ### Threshold Tuning Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Demonstrates how to adjust the confidence threshold to filter detection results. ```python from autodistill_grounded_sam_2 import GroundedSAM2 # Initialize with a specific threshold model = GroundedSAM2(threshold=0.2) # Or adjust after prediction if the model supports it (check predict method) # results = model.predict(image=image) # filtered_results = results[results.confidence > 0.4] ``` -------------------------------- ### Check Autodistill Grounded SAM 2 Installation Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Verify the installed versions of the Autodistill Grounded SAM 2 package, PyTorch, and CUDA availability. ```python from autodistill_grounded_sam_2 import __version__ print(f"Package version: {__version__}") import torch print(f"PyTorch version: {torch.__version__}") print(f"CUDA available: {torch.cuda.is_available()}") ``` -------------------------------- ### Checking Autodistill Cache Directory Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/04-errors.md Provides an example of checking the existence and calculating the size of the Autodistill cache directory. This is useful for managing pre-downloaded models. ```python import os from pathlib import Path cache_dir = Path.home() / ".cache" / "autodistill" if cache_dir.exists(): print(f"Cache size: {sum(f.stat().st_size for f in cache_dir.rglob('*') if f.is_file())}") ``` -------------------------------- ### Check Package Version Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Prints the currently installed version of the autodistill-grounded-sam-2 package. ```python from autodistill_grounded_sam_2 import __version__ print(__version__) ``` -------------------------------- ### Grounded SAM 2 Quickstart with Image Prediction and Labeling Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/README.md This Python script demonstrates how to use the GroundedSAM2 model for object detection. It defines an ontology for class mapping, runs inference on a single image, plots the results, and labels all images in a specified directory. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology from autodistill.utils import plot import cv2 # define an ontology to map class names to our Grounded SAM 2 prompt # the ontology dictionary has the format {caption: class} # where caption is the prompt sent to the base model, and class is the label that will # be saved for that caption in the generated annotations # then, load the model base_model = GroundedSAM2( ontology=CaptionOntology( { "person": "person", "shipping container": "shipping container", } ) ) # run inference on a single image results = base_model.predict("logistics.jpeg") plot( image=cv2.imread("logistics.jpeg"), classes=base_model.ontology.classes(), detections=results ) # label all images in a folder called `context_images` base_model.label("./context_images", extension=".jpeg") ``` -------------------------------- ### Simple Ontology Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md A straightforward ontology with one prompt for each distinct object class. ```python # Simple ontology: one prompt per class simple = CaptionOntology({ "person": "person", "dog": "dog" }) ``` -------------------------------- ### Custom Exception Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Illustrates handling custom exceptions like ValueError, RuntimeError, URLError, and FileNotFoundError. ```python try: # Code that might raise an error pass except ValueError as e: print(f"ValueError caught: {e}") except RuntimeError as e: print(f"RuntimeError caught: {e}") except FileNotFoundError as e: print(f"FileNotFoundError caught: {e}") except Exception as e: print(f"An unexpected error occurred: {e}") ``` -------------------------------- ### Development Dependencies Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Lists optional dependencies for development, including linting, formatting, testing, and packaging tools. Install these using pip with the '[dev]' extra. ```python extras_require={ "dev": ["flake8", "black==22.3.0", "isort", "twine", "pytest", "wheel"] } ``` ```bash pip install autodistill-grounded-sam-2[dev] ``` -------------------------------- ### Access Package Version Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/08-module-exports.md Retrieve the installed version of the autodistill_grounded_sam_2 package. This is useful for checking compatibility or reporting issues. ```python import autodistill_grounded_sam_2 print(autodistill_grounded_sam_2.__version__) # "0.1.0" ``` -------------------------------- ### API Reference: GroundedSAM2 Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/README.md Documentation for the main `GroundedSAM2` class, including its constructor, method signatures, return types, behavior specifications, and usage examples. ```APIDOC ## API Reference: GroundedSAM2 ### Description Provides documentation for the main `GroundedSAM2` class, detailing its constructor, method signatures, return types, behavior specifications, and complete usage examples. ### Method Signatures - `GroundedSAM2(...)` - Constructor - `GroundedSAM2.predict(...)` - Method for making predictions ### Return Types - Specifies the types and behavior of the data returned by the class methods. ### Usage Examples - Includes complete examples demonstrating how to use the `GroundedSAM2` class. ``` -------------------------------- ### Auto-labeling Directories Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Conceptual example for auto-labeling images within a directory. This would involve iterating through files and applying the model. ```python import os import cv2 from autodistill_grounded_sam_2 import GroundedSAM2 model = GroundedSAM2(ontology=["object"]) input_dir = "path/to/images" output_dir = "path/to/labels" for filename in os.listdir(input_dir): if filename.endswith(('.jpg', '.png')): image_path = os.path.join(input_dir, filename) image = cv2.imread(image_path) if image is not None: results = model.predict(image=image) # Save results to output_dir in a desired format (e.g., YOLO, COCO) print(f"Processed {filename}") ``` -------------------------------- ### Initialize GroundedSAM2 with Grounding DINO Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/01-api-reference-grounded-sam-2.md Initializes the GroundedSAM2 model using the Grounding DINO model with custom confidence thresholds. This approach requires specific setup for Grounding DINO. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology # Create ontology mapping captions to class names ontology = CaptionOntology({ "person": "person", "shipping container": "shipping container", "dog": "dog" }) # Or use Grounding DINO with custom thresholds model = GroundedSAM2( ontology=ontology, model="Grounding DINO", grounding_dino_box_threshold=0.4, grounding_dino_text_threshold=0.3 ) ``` -------------------------------- ### Clone SAM 2 Repository Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Use this command to clone the SAM 2 repository. Ensure Git version 2.0 or higher is installed. ```bash git clone https://github.com/facebookresearch/segment-anything-2.git ``` -------------------------------- ### Switch to Florence 2 Model Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Switch to the Florence 2 model if Grounding DINO is not providing satisfactory results. This requires the `autodistill-florence-2` package to be installed. ```python model = GroundedSAM2(ontology=ontology, model="Florence 2") ``` -------------------------------- ### Hierarchical Ontology Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md Configures an ontology with both broad categories and specific sub-classes, enabling detection at different levels of detail. ```python # Hierarchical ontology: broad and specific classes hierarchical = CaptionOntology({ "vehicle": "vehicle", "car": "car", "truck": "truck", "motorcycle": "motorcycle" }) ``` -------------------------------- ### Batch Image Processing with GroundedSAM2 Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/07-usage-examples.md Process multiple images from a directory using GroundedSAM2. This example iterates through JPG files in a specified directory and handles potential errors during processing. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology import os from pathlib import Path ontology = CaptionOntology({ "person": "person", "dog": "dog" }) model = GroundedSAM2(ontology=ontology) # Process all images in a directory image_dir = Path("path/to/images") for image_path in image_dir.glob("*.jpg"): try: detections = model.predict(str(image_path)) print(f"{image_path.name}: {len(detections)} objects detected") except Exception as e: print(f"Error processing {image_path.name}: {e}") ``` -------------------------------- ### Initialize GroundedSAM2 with Default Model Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/01-api-reference-grounded-sam-2.md Initializes the GroundedSAM2 model using the default Florence 2 predictor. Ensure you have the necessary dependencies installed for Florence 2. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology # Create ontology mapping captions to class names ontology = CaptionOntology({ "person": "person", "shipping container": "shipping container", "dog": "dog" }) # Initialize with default Florence 2 model model = GroundedSAM2(ontology=ontology) ``` -------------------------------- ### Using Grounding DINO Model with GroundedSAM2 Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/07-usage-examples.md Utilize the Grounding DINO model for detection, which is suitable for simple class names and explicit multi-prompt detection. This example uses default thresholds. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology ontology = CaptionOntology({ "person": "person", "dog": "dog", "cat": "cat" }) # Grounding DINO mode with default thresholds model = GroundedSAM2( ontology=ontology, model="Grounding DINO" ) detections = model.predict("image.jpg") ``` -------------------------------- ### Combine Detections from Multiple Prompts in GroundingSAM2 Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/02-api-reference-helpers.md This example shows how `combine_detections` is used within the GroundingSAM2 project to merge detection results obtained from multiple text prompts. It assigns class IDs sequentially based on the number of prompts, ensuring each prompt's detections are distinctly categorized. ```python from autodistill_grounded_sam_2.helpers import combine_detections # Assuming ontology and grounding_dino_model are defined elsewhere # ontology.prompts() returns a list of text descriptions # grounding_dino_model.predict_with_classes(...) performs detection detections_list = [] for i, description in enumerate(ontology.prompts()): detections = grounding_dino_model.predict_with_classes( image=image, classes=[description], ... ) detections_list.append(detections) # Combine with class IDs matching prompt order combined = combine_detections( detections_list, overwrite_class_ids=range(len(detections_list)) ) ``` -------------------------------- ### Calculate Total Disk Space Needed Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Estimates the total disk space required for installation, cached models, and working space. This calculation helps in planning storage requirements. ```text Installation: 1 GB (dependencies) + Models: 4 GB (cached on first use) + Working space: 2 GB (recommended) = Total needed: 7 GB minimum ``` -------------------------------- ### Load Grounding DINO Model Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/02-api-reference-helpers.md Loads the Grounding DINO model, downloading it if it's not found in the cache. This model is ready for inference on the configured device. Ensure necessary dependencies are installed to avoid runtime errors. ```python from autodistill_grounded_sam_2.helpers import load_grounding_dino # Load model (downloads on first run) model = load_grounding_dino() # Model can now be used for detection # model.predict_with_classes(image, classes, box_threshold, text_threshold) ``` -------------------------------- ### Standard GroundedSAM2 Initialization Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/08-module-exports.md Demonstrates initializing the GroundedSAM2 model with a CaptionOntology. This is the standard way to set up the model for use. ```python ontology = CaptionOntology({"person": "person"}) model = GroundedSAM2(ontology=ontology) ``` -------------------------------- ### Advanced GroundedSAM2 Initialization and Helper Usage Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/08-module-exports.md Shows how to initialize the GroundedSAM2 model and also load separate components like GroundingDINO and SAM using helper functions. This is useful when you need to work with individual model parts. ```python ontology = CaptionOntology({"person": "person"}) model = GroundedSAM2(ontology=ontology) # Or work with models directly dino = load_grounding_dino() sam2 = load_SAM() ``` -------------------------------- ### Initialize GroundedSAM2 with Grounding DINO Model Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/03-types.md Instantiate the GroundedSAM2 model using the Grounding DINO model for grounding. The ontology must be provided. ```python model2 = GroundedSAM2(ontology=ontology, model="Grounding DINO") ``` -------------------------------- ### Detection Result Processing Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Example of processing the Detections object returned by the predict method. ```python from supervision import Detections # Assuming 'results' is a Detections object from model.predict() for i in range(len(results)): print(f"Detection {i}:") print(f" Bounding Box: {results.xyxy[i]}") print(f" Confidence: {results.confidence[i]}") print(f" Class ID: {results.class_id[i]}") if results.mask is not None: print(f" Mask shape: {results.mask[i].shape}") ``` -------------------------------- ### Manual Pre-download for Grounding DINO Weights Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md Manually download the Grounding DINO weights to pre-populate the cache. This ensures the model is available without an automatic download on first use. ```bash # Grounding DINO weights wget -O ~/.cache/autodistill/groundingdino/groundingdino_swint_ogc.pth \ https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth ``` -------------------------------- ### Initialize GroundedSAM2 with Grounding DINO (Default Thresholds) Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Use Grounding DINO for explicit multi-prompt detection with default confidence thresholds. ```python model = GroundedSAM2(ontology=ontology, model="Grounding DINO") ``` -------------------------------- ### Configure GroundedSAM2 with Florence 2 Defaults Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md Instantiate GroundedSAM2 using the 'Florence 2' model to leverage its balanced default detection settings without manual threshold tuning. ```python model = GroundedSAM2( ontology=ontology, model="Florence 2" ) ``` -------------------------------- ### Initialize GroundedSAM2 with Florence 2 Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Use Florence 2 for semantic understanding and fine-grained descriptions. This is the default model. ```python model = GroundedSAM2(ontology=ontology) # or explicitly: model = GroundedSAM2(ontology=ontology, model="Florence 2") ``` -------------------------------- ### Configure Pip Proxy Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Configure pip to use a proxy server if you are behind a firewall. This command installs a package through a proxy. ```bash pip install --proxy [user:passwd@]proxy.server:port autodistill-grounded-sam-2 ``` -------------------------------- ### Model Initialization with Error Handling Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/07-usage-examples.md Demonstrates how to initialize the GroundedSAM2 model with error handling for common issues like invalid configurations or model loading failures. It catches specific exceptions and provides informative error messages. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology ontology = CaptionOntology({"person": "person"}) try: model = GroundedSAM2( ontology=ontology, model="Florence 2" ) print("Model loaded successfully") except ValueError as e: print(f"Invalid configuration: {e}") except RuntimeError as e: print(f"Model loading failed: {e}") except Exception as e: print(f"Unexpected error: {e}") ``` -------------------------------- ### Manual Pre-download for Grounding DINO Config Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md Manually download the Grounding DINO configuration file to pre-populate the cache. This ensures the model is available without an automatic download on first use. ```bash # Pre-download models manually # Grounding DINO config curl -o ~/.cache/autodistill/groundingdino/GroundingDINO_SwinT_OGC.py \ https://raw.githubusercontent.com/roboflow/GroundingDINO/main/groundingdino/config/GroundingDINO_SwinT_OGC.py ``` -------------------------------- ### Initialize GroundedSAM2 with Florence 2 Model Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/03-types.md Instantiate the GroundedSAM2 model using the Florence 2 model for grounding. Ensure the ontology is defined prior to instantiation. ```python model1 = GroundedSAM2(ontology=ontology, model="Florence 2") ``` -------------------------------- ### Detailed Ontology Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md An ontology using multiple descriptive prompts to identify variations of the same object class, allowing for more nuanced detection. ```python # Detailed ontology: multiple descriptive prompts per concept detailed = CaptionOntology({ "person wearing clothes": "person", "dog with collar": "dog", "red shipping container": "shipping_container", "blue shipping container": "shipping_container" }) ``` -------------------------------- ### Initialize GroundedSAM2 with Grounding DINO (Custom Thresholds) Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md Configure GroundedSAM2 with Grounding DINO and custom detection thresholds for bounding boxes and text-image alignment. This provides more control over filtering. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology ontology = CaptionOntology({ "person": "person", "dog": "dog", "car": "car" }) # Advanced configuration with custom thresholds model = GroundedSAM2( ontology=ontology, model="Grounding DINO", grounding_dino_box_threshold=0.4, grounding_dino_text_threshold=0.3 ) ``` -------------------------------- ### List Outdated Dependencies Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Check for available updates to installed Python packages. This command lists all packages that have newer versions available on PyPI. ```bash pip list --outdated ``` -------------------------------- ### Initialize GroundedSAM2 with Grounding DINO (Conservative) Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Use Grounding DINO with higher confidence thresholds for box and text detection when false positives are costly. ```python model = GroundedSAM2( ontology=ontology, model="Grounding DINO", grounding_dino_box_threshold=0.50, grounding_dino_text_threshold=0.40 ) ``` -------------------------------- ### Mask Post-Processing Example Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/06-architecture-overview.md Add morphological operations to the mask post-processing step after line 78 in the predict() function to refine generated masks. ```python # Add morphological operations kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5)) masks[index] = cv2.morphologyEx(masks[index], cv2.MORPH_CLOSE, kernel) ``` -------------------------------- ### Initialize GroundedSAM2 with Grounding DINO (Basic) Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md Initialize GroundedSAM2 using the Grounding DINO model. This basic configuration uses default thresholds. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology ontology = CaptionOntology({ "person": "person", "dog": "dog", "car": "car" }) # Basic configuration model = GroundedSAM2( ontology=ontology, model="Grounding DINO" ) ``` -------------------------------- ### GroundedSAM2 Constructor Parameters Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Initialize the GroundedSAM2 model. Specify the ontology, optionally choose between 'Florence 2' or 'Grounding DINO' models, and adjust detection and text thresholds. ```python GroundedSAM2( ontology: CaptionOntology, model: str = "Florence 2", grounding_dino_box_threshold: float = 0.35, grounding_dino_text_threshold: float = 0.25 ) ``` -------------------------------- ### Dependency Initialization Timeline Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/06-architecture-overview.md Outlines the sequence of events during the initialization of the GroundedSAM2 module, from module import to the first predict() call. ```text 1. Module import: - Environment variables set - DEVICE determined (CUDA if available, else CPU) - SamPredictor loaded via load_SAM() (line 21) 2. GroundedSAM2 instantiation: - If Florence 2: Creates Florence2(ontology) - If Grounding DINO: Calls load_grounding_dino() and caches result 3. First predict() call: - Image loaded via load_image() - Grounding model inference - SAM 2 inference per detection ``` -------------------------------- ### Initialize GroundedSAM2 with Grounding DINO (Permissive) Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Use Grounding DINO with lower confidence thresholds for box and text detection when missing objects is worse than false positives. ```python model = GroundedSAM2( ontology=ontology, model="Grounding DINO", grounding_dino_box_threshold=0.25, grounding_dino_text_threshold=0.15 ) ``` -------------------------------- ### Initialize GroundedSAM2 with Ontology Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Initializes the GroundedSAM2 model with a custom ontology for object detection. Ensure necessary imports are present. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology ontology = CaptionOntology({ "person": "person", "dog": "dog", "car": "car" }) model = GroundedSAM2(ontology=ontology) ``` -------------------------------- ### Get Class Names Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Retrieve a list of all class names available in the model's ontology. This can be used to map class IDs to human-readable names. ```python class_names = list(model.ontology.classes()) for i, name in enumerate(class_names): print(f"{i}: {name}") ``` -------------------------------- ### Check for Known Dependency Vulnerabilities Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/09-dependencies.md Use `pip-audit` to scan installed Python packages for known security vulnerabilities. This is crucial for maintaining a secure development environment. ```bash # Check for known vulnerabilities pip-audit ``` -------------------------------- ### Performance Optimization Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Tips for optimizing performance, such as using GPU acceleration and appropriate model caching. ```python # Ensure model is loaded on the correct device (e.g., 'cuda:0') model = GroundedSAM2(device="cuda:0") # Utilize model caching by setting model_cache_dir model = GroundedSAM2(model_cache_dir="~/.cache/autodistill/") ``` -------------------------------- ### Import Core Components Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Import the main GroundedSAM2 class, CaptionOntology for defining detection targets, and supervision for handling detection results. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology import supervision as sv from autodistill_grounded_sam_2.helpers import ( load_grounding_dino, load_SAM, combine_detections ) ``` -------------------------------- ### Handle subprocess and URLError during SAM loading Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/04-errors.md Handles errors from git clone, pip install, or checkpoint downloads for the SAM model. Includes network and dependency checks. ```python from autodistill_grounded_sam_2.helpers import load_SAM import subprocess import urllib.error try: predictor = load_SAM() except subprocess.CalledProcessError as e: print(f"Git clone or pip install failed: {e}") # Check network connectivity and disk space except urllib.error.URLError as e: print(f"Failed to download SAM2 checkpoint: {e}") # Retry with network check except RuntimeError as e: print(f"SAM2 model build failed: {e}") # Check dependencies and CUDA ``` -------------------------------- ### Predict Detections Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Use the predict method to get object detections from an image. The input can be an image path, numpy array, or cv2 image. The method returns supervision.Detections. ```python detections: sv.Detections = model.predict(input: Any) ``` -------------------------------- ### Import GroundedSAM2 Class Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/08-module-exports.md Import the main GroundedSAM2 class for object detection and segmentation. This is the primary entry point for using the library. ```python from autodistill_grounded_sam_2 import GroundedSAM2 ``` -------------------------------- ### Adjusting Grounding DINO Model Thresholds Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/04-errors.md Demonstrates how to adjust the `grounding_dino_box_threshold` and `grounding_dino_text_threshold` parameters when initializing the GroundedSAM2 model. Higher thresholds lead to fewer, more conservative detections. ```python from autodistill_grounded_sam_2 import GroundedSAM2 # Higher thresholds = fewer detections (more conservative) model = GroundedSAM2( ontology=ontology, model="Grounding DINO", grounding_dino_box_threshold=0.5, # Increased from 0.35 grounding_dino_text_threshold=0.4 # Increased from 0.25 ) ``` -------------------------------- ### Filtering Detections by Class Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/07-usage-examples.md Filter detection results to isolate detections belonging to a specific class ID. This example shows how to extract only 'person' detections. Ensure numpy is imported for array operations. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology import numpy as np ontology = CaptionOntology({ "person": "person", "dog": "dog", "cat": "cat" }) model = GroundedSAM2(ontology=ontology) detections = model.predict("image.jpg") # Keep only persons (class_id = 0) person_class_id = 0 if detections.class_id is not None: person_mask = detections.class_id == person_class_id person_detections = detections[person_mask] print(f"Found {len(person_detections)} persons") ``` -------------------------------- ### Access Detections Structure Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/10-quick-reference.md Access various attributes of the detections object, such as bounding boxes, masks, confidence scores, and class IDs. You can also get the number of detections or access individual detections by index or slice. ```python detections.xyxy # (N, 4) - Bounding boxes [x1, y1, x2, y2] detections.mask # (N, H, W) - Boolean segmentation masks detections.confidence # (N,) - Confidence scores (0.0-1.0) detections.class_id # (N,) - Integer class IDs (0 to num_classes-1) detections.tracker_id # (N,) - Tracker IDs (if tracking enabled) len(detections) # Number of detections detections[0] # Access first detection (returns single Detection) detections[0:5] # Slice detections (returns filtered Detections) ``` -------------------------------- ### Import GroundedSAM2 from Submodule Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/08-module-exports.md Importing directly from the submodule is possible but not recommended. ```python from autodistill_grounded_sam_2.grounded_sam_2 import GroundedSAM2 ``` -------------------------------- ### load_SAM Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/02-api-reference-helpers.md Loads the SAM2 (Segment Anything 2) model. It checks for cached versions, clones the repository if necessary, installs dependencies, downloads the model checkpoint, and initializes the predictor. This function is essential for setting up the SAM2 model for image segmentation tasks. ```APIDOC ## Function: `load_SAM` Loads SAM2 (Segment Anything 2) model from cache or builds it if not available. **Signature:** ```python def load_SAM() -> SAM2ImagePredictor ``` **Return Type:** `sam2.sam2_image_predictor.SAM2ImagePredictor` A SAM2 image predictor instance ready for inference. **Behavior:** 1. Saves current working directory 2. Constructs cache directory path: `~/.cache/autodistill/segment_anything_2/` 3. Constructs checkpoint path for `sam2_hiera_base_plus.pth` 4. Checks if SAM2 repository exists at `~/.cache/autodistill/segment_anything_2/segment-anything-2/` 5. If repository missing: - Clones SAM2 from: `https://github.com/facebookresearch/segment-anything-2.git` - Runs `pip install -e .` to install in editable mode 6. Adds SAM2 directory to `sys.path` 7. Downloads checkpoint if missing from: `https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_base_plus.pt` 8. Builds SAM2 predictor with: - Model config: `sam2_hiera_b+.yaml` (base+ variant) - Checkpoint: `sam2_hiera_base_plus.pth` 9. Restores original working directory 10. Returns initialized predictor **Raises:** | Error | Condition | |-------|-----------| | subprocess.CalledProcessError | Git clone or pip install fails | | urllib.error.URLError | Network error during checkpoint download | | FileNotFoundError | Unable to create cache directory | | RuntimeError | SAM2 build failure (missing config or checkpoint) | **Side Effects:** - Modifies `sys.path` (adds SAM2 installation directory) - Creates directories under `~/.cache/autodistill/` - May run git and pip commands - Changes working directory during execution **Example:** ```python from autodistill_grounded_sam_2.helpers import load_SAM # Load SAM2 (clones and builds on first run) predictor = load_SAM() # Set image and predict masks predictor.set_image(image_array) masks, scores, logits = predictor.predict(box=box_array, multimask_output=False) ``` ``` -------------------------------- ### Initialize GroundedSAM2 Model Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/README.md Instantiate the GroundedSAM2 class, specifying the ontology and optionally the grounding model and its thresholds. Defaults to Florence 2 model. ```python class GroundedSAM2(DetectionBaseModel): def __init__( self, ontology: CaptionOntology, model: str = "Florence 2", grounding_dino_box_threshold: float = 0.35, grounding_dino_text_threshold: float = 0.25 ) -> None ``` -------------------------------- ### Autodistill GroundedSAM2 File Index Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/README.md This table provides an index of the documentation files within the Autodistill GroundedSAM2 project. It lists each file, its approximate line count, and its primary purpose. ```markdown | File | Lines | Purpose | |------|-------|---------| | 01-api-reference-grounded-sam-2.md | ~200 | GroundedSAM2 class documentation | | 02-api-reference-helpers.md | ~250 | Helper functions documentation | | 03-types.md | ~200 | Type definitions and data structures | | 04-errors.md | ~300 | Error handling and exceptions | | 05-configuration.md | ~400 | Configuration options and setup | | 06-architecture-overview.md | ~350 | System design and data flow | | 07-usage-examples.md | ~500 | Production-ready code examples | | 08-module-exports.md | ~250 | Imports and module structure | | 09-dependencies.md | ~400 | Packages and requirements | | 10-quick-reference.md | ~300 | Quick lookup and cheat sheets | | README.md | ~400 | This file | **Total:** ~3,550 lines of technical reference ``` -------------------------------- ### Import Patterns Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Standard import patterns for using the Autodistill Grounded SAM 2 library. ```python import autodistill_grounded_sam_2 from autodistill_grounded_sam_2.helpers import load_SAM from supervision import Detections ``` -------------------------------- ### Manual Pre-download for SAM 2 Checkpoint Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md Manually download the SAM 2 checkpoint to pre-populate the cache. This ensures the model is available without an automatic download on first use. ```bash # SAM 2 checkpoint wget -O ~/.cache/autodistill/segment_anything_2/sam2_hiera_base_plus.pth \ https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_base_plus.pt ``` -------------------------------- ### Model Selection: Florence 2 vs Grounding DINO Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/MANIFEST.txt Autodistill Grounded SAM 2 supports both Florence 2 (default) and Grounding DINO models. Select the desired model during initialization. ```python # Using Florence 2 (default) model_florence = GroundedSAM2() # Using Grounding DINO (requires loading helper) from autodistill_grounded_sam_2.helpers import load_grounding_dino model_dino = load_grounding_dino() ``` -------------------------------- ### Define Scene-Specific Ontologies Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/07-usage-examples.md Create different ontologies tailored to specific environments like retail or warehouses. Use the appropriate ontology when initializing the model. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology # Customize ontology for your domain retail_ontology = CaptionOntology({ "person": "person", "shopping cart": "cart", "product shelf": "shelf", "cash register": "register", "price tag": "tag" }) warehouse_ontology = CaptionOntology({ "person": "person", "forklift": "equipment", "pallet": "pallet", "box": "box", "safety cone": "hazard" }) # Use appropriate ontology for scene retail_model = GroundedSAM2(ontology=retail_ontology) warehouse_model = GroundedSAM2(ontology=warehouse_ontology) ``` -------------------------------- ### Import Helper Functions Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/08-module-exports.md Import helper functions like combine_detections if they are needed. ```python from autodistill_grounded_sam_2.helpers import combine_detections ``` -------------------------------- ### GroundedSAM2 Constructor Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/01-api-reference-grounded-sam-2.md Initializes a GroundedSAM2 model for object detection and segmentation. You can specify the ontology, the grounding model to use (Florence 2 or Grounding DINO), and model-specific thresholds. ```APIDOC ## GroundedSAM2 Constructor ### Description Initializes a GroundedSAM2 model for object detection and segmentation. You can specify the ontology, the grounding model to use (Florence 2 or Grounding DINO), and model-specific thresholds. ### Signature ```python def __init__( self, ontology: CaptionOntology, model: str = "Florence 2", grounding_dino_box_threshold: float = 0.35, grounding_dino_text_threshold: float = 0.25 ) -> None ``` ### Parameters #### Parameters - **ontology** (CaptionOntology) - Required - Ontology mapping class names to prompts for detection. Maps caption strings to class labels. - **model** (str) - Optional - Default: "Florence 2" - Grounding model to use for detection. Supported values: "Florence 2", "Grounding DINO". - **grounding_dino_box_threshold** (float) - Optional - Default: 0.35 - Confidence threshold for bounding boxes when using Grounding DINO model. Range: 0.0-1.0. - **grounding_dino_text_threshold** (float) - Optional - Default: 0.25 - Confidence threshold for text matching when using Grounding DINO model. Range: 0.0-1.0. ### Raises - **ValueError**: When `model` parameter is not in SUPPORTED_GROUNDING_MODELS. ### Example ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology ontology = CaptionOntology({ "person": "person", "shipping container": "shipping container", "dog": "dog" }) # Initialize with default Florence 2 model model = GroundedSAM2(ontology=ontology) # Or use Grounding DINO with custom thresholds model = GroundedSAM2( ontology=ontology, model="Grounding DINO", grounding_dino_box_threshold=0.4, grounding_dino_text_threshold=0.3 ) ``` ``` -------------------------------- ### Basic Ontology Configuration Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/05-configuration.md Defines a basic ontology with a few object classes. Keys are descriptive prompts, and values are the output class labels. ```python from autodistill.detection import CaptionOntology ontology = CaptionOntology({ "caption_or_prompt": "class_label", "person": "person", "shipping container": "shipping_container", "red vehicle": "vehicle" }) ``` -------------------------------- ### Comparing Florence 2 and Grounding DINO Results Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/07-usage-examples.md Run predictions on the same image using both Florence 2 and Grounding DINO models to compare their detection results. This helps in understanding the differences in their output. ```python from autodistill_grounded_sam_2 import GroundedSAM2 from autodistill.detection import CaptionOntology ontology = CaptionOntology({"person": "person", "dog": "dog"}) # Test both models on same image florence = GroundedSAM2(ontology=ontology, model="Florence 2") dino = GroundedSAM2(ontology=ontology, model="Grounding DINO") florence_results = florence.predict("image.jpg") dino_results = dino.predict("image.jpg") print(f"Florence 2: {len(florence_results)} detections") print(f"Grounding DINO: {len(dino_results)} detections") ``` -------------------------------- ### Quick Reference Source: https://github.com/autodistill/autodistill-grounded-sam-2/blob/main/_autodocs/README.md Provides a quick lookup for API, common operations, troubleshooting, and code snippets. ```APIDOC ## Quick Reference ### Description Offers a quick reference guide for fast API lookup, common operations, troubleshooting, and useful code snippets. ### Fast API Lookup - A concise index of API elements. ### Common Operations - Lists frequently used operations. ### Troubleshooting Guide - Guidance for resolving common issues. ### Cheat Sheets - Provides cheat sheets for quick reference. ### Code Snippets - Collection of useful code snippets for various tasks. ```