### Install Dependencies Source: https://github.com/nvlabs/mambavision/blob/main/README.md Installs project dependencies from the requirements.txt file. Ensure PyTorch is installed beforehand. ```bash pip install -r requirements.txt ``` -------------------------------- ### Install MMPretrain Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/convnext/README.md Install the MMPretrain library, which is required for using ConvNeXt backbones. Ensure you are using version 1.0.0rc7 or later. ```shell pip install mmpretrain>=1.0.0rc7 ``` -------------------------------- ### Install MambaVision Package Source: https://github.com/nvlabs/mambavision/blob/main/README.md Install the MambaVision library using pip. This is a prerequisite for using the models directly from the pip package. ```bash pip install mambavision ``` -------------------------------- ### Install MMPreTrain for ConvNeXt Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/configs/convnext/README.md Install the MMPreTrain library to utilize ConvNeXt backbones for downstream tasks. This is a prerequisite for using the provided object detection configurations. ```shell pip install mmpretrain ``` -------------------------------- ### Install Object Detection Dependencies Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/README.md Installs the necessary MambaVision and MMDetection framework packages. It's recommended to pin these versions for reproducibility. ```bash pip install \ mmengine==0.10.1 \ mmcv==2.1.0 \ opencv-python-headless \ mmdet==3.3.0 \ mmsegmentation==1.2.2 \ mmpretrain==1.2.0 ``` -------------------------------- ### Create MambaVision Model from Pip Package Source: https://github.com/nvlabs/mambavision/blob/main/README.md Create a pretrained MambaVision model with default hyperparameters directly from the installed pip package. ```python >>> from mambavision import create_model ``` -------------------------------- ### Install Semantic Segmentation Dependencies Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/README.md Installs the necessary packages for semantic segmentation using MambaVision, building upon the MMSegmentation framework. It's recommended to pin these versions in a requirements.txt file for reproducibility. ```bash pip install \ mmengine==0.10.1 \ mmcv==2.1.0 \ opencv-python-headless \ mmsegmentation==1.2.2 \ mmdet==3.3.0 \ mmpretrain==1.2.0 ``` -------------------------------- ### COCO Dataset Download and Extraction Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/README.md Prepare the COCO dataset by downloading and extracting the necessary image and annotation files. Ensure you are in the MMDetection root directory before starting. ```bash cd mkdir -p data/coco && cd data/coco wget http://images.cocodataset.org/zips/train2017.zip wget http://images.cocodataset.org/zips/val2017.zip wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip unzip train2017.zip unzip val2017.zip unzip annotations_trainval2017.zip ``` -------------------------------- ### Cascade Mask R-CNN with ConvNeXt-T Configuration Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/configs/convnext/README.md Configuration for Cascade Mask R-CNN with a ConvNeXt-T backbone. This setup includes FP16 precision, multi-scale cropping, and a 3x training schedule, achieving 50.3 box AP and 43.6 mask AP. ```text config: ./cascade-mask-rcnn_convnext-t-p4-w7_fpn_4conv1fc-giou_amp-ms-crop-3x_coco.py model: https://download.openmmlab.com/mmdetection/v2.0/convnext/cascade_mask_rcnn_convnext-t_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco/cascade_mask_rcnn_convnext-t_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco_20220509_204200-8f07c40b.pth log: https://download.openmmlab.com/mmdetection/v2.0/convnext/cascade_mask_rcnn_convnext-t_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco/cascade_mask_rcnn_convnext-t_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco_20220509_204200.log.json ``` -------------------------------- ### Validate Pip Model Source: https://github.com/nvlabs/mambavision/blob/main/README.md Runs validation for a Mamba Vision model installed via pip. Requires specifying the model name, data directory, and batch size. ```bash python validate_pip_model.py --model mamba_vision_T --data_dir=$DATA_PATH --batch-size $BS ``` -------------------------------- ### Download ADE20K Dataset Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/README.md Commands to download and unzip the ADE20K dataset. Ensure you are in the correct root directory. ```bash cd mkdir -p data/ade20k && cd data/ade20k wget http://data.csail.mit.edu/places/ADEchallenge/ADEChallengeData2016.zip unzip ADEChallengeData2016.zip ``` -------------------------------- ### Multi-GPU Training Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/README.md Launches multi-GPU training for object detection models using a specified configuration file. This command is intended for use with Slurm. ```bash srun --gres=gpu:8 python tools/train.py configs/mamba_vision/.py ``` -------------------------------- ### UperNet Model Download (Pascal VOC 2012, 20k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md Download link for the pre-trained UperNet model with a ResNet-50 backbone, trained for 20,000 iterations on Pascal VOC 2012 with augmentation. ```bash https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_20k_voc12aug/upernet_r50_512x512_20k_voc12aug_20200617_165330-5b5890a7.pth ``` -------------------------------- ### UperNet Model Download (Pascal VOC 2012, 40k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md Download link for the pre-trained UperNet model with a ResNet-50 backbone, trained for 40,000 iterations on Pascal VOC 2012 with augmentation. ```bash https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_40k_voc12aug/upernet_r50_512x512_40k_voc12aug_20200613_162257-ca9bcc6b.pth ``` -------------------------------- ### UperNet Configuration for Pascal VOC 2012 (40k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md This configuration is for the UperNet model with a ResNet-50 backbone, trained for 40,000 iterations on Pascal VOC 2012 with augmentation, using a 512x512 crop size. ```python upernet_r50_4xb4-40k_voc12aug-512x512.py ``` -------------------------------- ### UperNet Configuration for Pascal VOC 2012 (20k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md This configuration is for the UperNet model with a ResNet-50 backbone, trained for 20,000 iterations on Pascal VOC 2012 with augmentation, using a 512x512 crop size. ```python upernet_r50_4xb4-20k_voc12aug-512x512.py ``` -------------------------------- ### UperNet Model Download (Pascal VOC 2012, R-101, 20k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md Download link for the pre-trained UperNet model with a ResNet-101 backbone, trained for 20,000 iterations on Pascal VOC 2012 with augmentation. ```bash https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_20k_voc12aug/upernet_r101_512x512_20k_voc12aug_20200617_165629-f14e7f27.pth ``` -------------------------------- ### UperNet Model Download (Pascal VOC 2012, R-101, 40k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md Download link for the pre-trained UperNet model with a ResNet-101 backbone, trained for 40,000 iterations on Pascal VOC 2012 with augmentation. ```bash https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_40k_voc12aug/upernet_r101_512x512_40k_voc12aug_20200613_163549-e26476ac.pth ``` -------------------------------- ### UperNet Configuration for Pascal VOC 2012 (R-101, 40k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md This configuration is for the UperNet model with a ResNet-101 backbone, trained for 40,000 iterations on Pascal VOC 2012 with augmentation, using a 512x512 crop size. ```python upernet_r101_4xb4-40k_voc12aug-512x512.py ``` -------------------------------- ### UperNet Configuration for Pascal VOC 2012 (R-101, 20k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md This configuration is for the UperNet model with a ResNet-101 backbone, trained for 20,000 iterations on Pascal VOC 2012 with augmentation, using a 512x512 crop size. ```python upernet_r101_4xb4-20k_voc12aug-512x512.py ``` -------------------------------- ### UperNet Log File (Pascal VOC 2012, 20k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md Log file for the UperNet model with a ResNet-50 backbone, trained for 20,000 iterations on Pascal VOC 2012 with augmentation. ```json https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_20k_voc12aug/upernet_r50_512x512_20k_voc12aug_20200617_165330.log.json ``` -------------------------------- ### UperNet Log File (Pascal VOC 2012, 40k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md Log file for the UperNet model with a ResNet-50 backbone, trained for 40,000 iterations on Pascal VOC 2012 with augmentation. ```json https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_40k_voc12aug/upernet_r50_512x512_40k_voc12aug_20200613_162257.log.json ``` -------------------------------- ### Single-GPU Training Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/README.md Execute this command for quick tests or debugging on a single GPU. Set the CUDA_VISIBLE_DEVICES environment variable. ```bash # single‑GPU training env CUDA_VISIBLE_DEVICES=0 python tools/train.py ${CONFIG} ``` -------------------------------- ### UperNet Log File (Pascal VOC 2012, R-101, 20k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md Log file for the UperNet model with a ResNet-101 backbone, trained for 20,000 iterations on Pascal VOC 2012 with augmentation. ```json https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_20k_voc12aug/upernet_r101_512x512_20k_voc12aug_20200617_165629.log.json ``` -------------------------------- ### UperNet Log File (Pascal VOC 2012, R-101, 40k schedule) Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md Log file for the UperNet model with a ResNet-101 backbone, trained for 40,000 iterations on Pascal VOC 2012 with augmentation. ```json https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_40k_voc12aug/upernet_r101_512x512_40k_voc12aug_20200613_163549.log.json ``` -------------------------------- ### Create Mamba Vision Model Source: https://github.com/nvlabs/mambavision/blob/main/README.md Instantiates a Mamba Vision model with pretrained weights. Specify the model name and the path to the pretrained weights. ```python model = create_model('mamba_vision_T', pretrained=True, model_path="/tmp/mambavision_tiny_1k.pth.tar") ``` -------------------------------- ### Citation for UperNet Paper Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/upernet/README.md BibTeX entry for the paper 'Unified perceptual parsing for scene understanding' by Xiao et al., which introduced the UperNet architecture. ```bibtex @inproceedings{xiao2018unified, title={Unified perceptual parsing for scene understanding}, author={Xiao, Tete and Liu, Yingcheng and Zhou, Bolei and Jiang, Yuning and Sun, Jian}, booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, pages={418--434}, year={2018} } ``` -------------------------------- ### Single-GPU Inference Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/README.md Use this command for quick evaluation or debugging on a single GPU. Specify the configuration and checkpoint files. ```bash # single‑GPU inference env CUDA_VISIBLE_DEVICES=0 \ python tools/test.py \ configs/mamba_vision/.py \ .pth \ ``` -------------------------------- ### Single-GPU Training Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/README.md Runs object detection training on a single GPU, suitable for quick tests or debugging. Ensure the CUDA_VISIBLE_DEVICES environment variable is set correctly. ```bash env CUDA_VISIBLE_DEVICES=0 python tools/train.py ${CONFIG} ``` -------------------------------- ### Load MambaVision Model from Hugging Face Source: https://github.com/nvlabs/mambavision/blob/main/README.md Import and load a pretrained MambaVision model for image classification from Hugging Face. Ensure 'trust_remote_code' is set to True. ```python >>> from transformers import AutoModelForImageClassification >>> model = AutoModelForImageClassification.from_pretrained("nvidia/MambaVision-T-1K", trust_remote_code=True) ``` -------------------------------- ### Convert Swin Transformer Model Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/configs/swin/README.md Use this script to convert model keys from the official Swin Transformer repository to the MMSegmentation style. Specify the path to the original pre-trained model and the desired storage path for the converted model. ```shell python tools/model_converters/swin2mmseg.py ${PRETRAIN_PATH} ${STORE_PATH} ``` ```shell python tools/model_converters/swin2mmseg.py https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pth pretrain/swin_base_patch4_window7_224.pth ``` -------------------------------- ### Mask R-CNN Swin-T Configuration (3x LR schedule, multi-scale crop) Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/configs/swin/README.md Configuration for Mask R-CNN with Swin-T backbone, ImageNet-1K pre-training, 3x LR schedule, and multi-scale cropping enabled. This configuration aims for improved performance by using more training epochs and varied input scales. ```python mask-rcnn_swin-t-p4-w7_fpn_ms-crop-3x_coco.py ``` -------------------------------- ### End-to-End Image Classification with MambaVision (Hugging Face) Source: https://github.com/nvlabs/mambavision/blob/main/README.md Perform image classification using a MambaVision model loaded from Hugging Face. This includes image preparation, model inference, and predicting the class. ```python from transformers import AutoModelForImageClassification from PIL import Image from timm.data.transforms_factory import create_transform import requests model = AutoModelForImageClassification.from_pretrained("nvidia/MambaVision-T-1K", trust_remote_code=True) # eval mode for inference model.cuda().eval() # prepare image for the model url = 'http://images.cocodataset.org/val2017/000000020247.jpg' image = Image.open(requests.get(url, stream=True).raw) input_resolution = (3, 224, 224) # MambaVision supports any input resolutions transform = create_transform(input_size=input_resolution, is_training=False, mean=model.config.mean, std=model.config.std, crop_mode=model.config.crop_mode, crop_pct=model.config.crop_pct) inputs = transform(image).unsqueeze(0).cuda() # model inference outputs = model(inputs) logits = outputs['logits'] predicted_class_idx = logits.argmax(-1).item() print("Predicted class:", model.config.id2label[predicted_class_idx]) ``` -------------------------------- ### Multi-GPU Inference Script Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/README.md Execute the provided test script for high-throughput evaluation using 8 GPUs. This is suitable for distributed training or large-scale testing. ```bash # multi‑GPU inference (8 GPUs) bash tools/test.sh ``` -------------------------------- ### Configure MMSEGMENTATION for ADE20K Source: https://github.com/nvlabs/mambavision/blob/main/semantic_segmentation/README.md Python code snippet to configure the MMSEGMENTATION data loader for the ADE20K dataset. Set the data root and specify image/annotation directories. ```python data_root = 'data/ade20k/' data = dict( train=dict( type='ADE20KDataset', data_root=data_root, img_dir='images/training', ann_dir='annotations/training'), val=dict( type='ADE20KDataset', data_root=data_root, img_dir='images/validation', ann_dir='annotations/validation'), test=dict( type='ADE20KDataset', data_root=data_root, img_dir='images/validation', ann_dir='annotations/validation') ) ``` -------------------------------- ### Mask R-CNN Swin-T Configuration (3x LR schedule, multi-scale crop, FP16) Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/configs/swin/README.md Configuration for Mask R-CNN with Swin-T backbone, ImageNet-1K pre-training, 3x LR schedule, multi-scale cropping, and FP16 enabled for faster training and reduced memory usage. This is an optimized configuration for performance and efficiency. ```python mask-rcnn_swin-t-p4-w7_fpn_amp-ms-crop-3x_coco.py ``` -------------------------------- ### Mask R-CNN Swin-S Configuration (3x LR schedule, multi-scale crop, FP16) Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/configs/swin/README.md Configuration for Mask R-CNN with Swin-S backbone, ImageNet-1K pre-training, 3x LR schedule, multi-scale cropping, and FP16 enabled. This configuration uses a larger Swin backbone (Swin-S) for potentially higher accuracy. ```python mask-rcnn_swin-s-p4-w7_fpn_amp-ms-crop-3x_coco.py ``` -------------------------------- ### Single-GPU Inference Command Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/README.md Run inference on a single GPU for debugging or quick evaluations. Specify the CUDA visible devices and point to your configuration and checkpoint files. ```bash # single‑GPU inference env CUDA_VISIBLE_DEVICES=0 \ python tools/test.py \ configs/mamba_vision/.py \ .pth \ --eval bbox segm ``` -------------------------------- ### Test Model with Dummy Image Source: https://github.com/nvlabs/mambavision/blob/main/README.md Tests the model's forward pass with a randomly generated image. Ensure the image and model are on the same device (e.g., CUDA). The output is a tensor of logits. ```python import torch image = torch.rand(1, 3, 512, 224).cuda() # place image on cuda model = model.cuda() # place model on cuda output = model(image) # output logit size is [1, 1000] ``` -------------------------------- ### Mask R-CNN with ConvNeXt-T Configuration Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/configs/convnext/README.md Configuration for Mask R-CNN with a ConvNeXt-T backbone, pre-trained on ImageNet-1K. This configuration uses a 3x training schedule, multi-scale cropping, and FP16 precision. ```text config: ./mask-rcnn_convnext-t-p4-w7_fpn_amp-ms-crop-3x_coco.py model: https://download.openmmlab.com/mmdetection/v2.0/convnext/mask_rcnn_convnext-t_p4_w7_fpn_fp16_ms-crop_3x_coco/mask_rcnn_convnext-t_p4_w7_fpn_fp16_ms-crop_3x_coco_20220426_154953-050731f4.pth log: https://download.openmmlab.com/mmdetection/v2.0/convnext/mask_rcnn_convnext-t_p4_w7_fpn_fp16_ms-crop_3x_coco/mask_rcnn_convnext-t_p4_w7_fpn_fp16_ms-crop_3x_coco_20220426_154953.log.json ``` -------------------------------- ### Cascade Mask R-CNN with ConvNeXt-S Configuration Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/configs/convnext/README.md Configuration for Cascade Mask R-CNN utilizing a ConvNeXt-S backbone. This model is trained with FP16, multi-scale cropping, and a 3x schedule, yielding 51.8 box AP and 44.8 mask AP. ```text config: ./cascade-mask-rcnn_convnext-s-p4-w7_fpn_4conv1fc-giou_amp-ms-crop-3x_coco.py model: https://download.openmmlab.com/mmdetection/v2.0/convnext/cascade_mask_rcnn_convnext-s_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco/cascade_mask_rcnn_convnext-s_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco_20220510_201004-3d24f5a4.pth log: https://download.openmmlab.com/mmdetection/v2.0/convnext/cascade_mask_rcnn_convnext-s_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco/cascade_mask_rcnn_convnext-s_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco_20220510_201004.log.json ``` -------------------------------- ### Feature Extraction with MambaVision (Hugging Face) Source: https://github.com/nvlabs/mambavision/blob/main/README.md Extract hierarchical multi-scale features from an image using a MambaVision model from Hugging Face. This snippet retrieves both averaged pool features and features from different model stages. ```python from transformers import AutoModel from PIL import Image from timm.data.transforms_factory import create_transform import requests model = AutoModel.from_pretrained("nvidia/MambaVision-T-1K", trust_remote_code=True) # eval mode for inference model.cuda().eval() # prepare image for the model url = 'http://images.cocodataset.org/val2017/000000020247.jpg' image = Image.open(requests.get(url, stream=True).raw) input_resolution = (3, 224, 224) # MambaVision supports any input resolutions transform = create_transform(input_size=input_resolution, is_training=False, mean=model.config.mean, std=model.config.std, crop_mode=model.config.crop_mode, crop_pct=model.config.crop_pct) inputs = transform(image).unsqueeze(0).cuda() # model inference out_avg_pool, features = model(inputs) print("Size of the averaged pool features:", out_avg_pool.size()) # torch.Size([1, 640]) print("Number of stages in extracted features:", len(features)) # 4 stages print("Size of extracted features in stage 1:", features[0].size()) # torch.Size([1, 80, 56, 56]) print("Size of extracted features in stage 4:", features[3].size()) # torch.Size([1, 640, 7, 7]) ``` -------------------------------- ### Mask R-CNN Swin-T Configuration (1x LR schedule) Source: https://github.com/nvlabs/mambavision/blob/main/object_detection/configs/swin/README.md Configuration for Mask R-CNN with Swin-T backbone, ImageNet-1K pre-training, 1x LR schedule, no multi-scale cropping, and FP16 disabled. This configuration is suitable for standard object detection tasks. ```python mask-rcnn_swin-t-p4-w7_fpn_1x_coco.py ``` -------------------------------- ### Evaluate MambaVision Models Source: https://github.com/nvlabs/mambavision/blob/main/README.md Evaluates MambaVision models on the ImageNet-1K validation set. Specify the model name, checkpoint path, data directory, and batch size. ```bash python validate.py \ --model --checkpoint --data_dir --batch-size