### Install Dependencies

Source: https://github.com/wangzt-halo/mv2dfusion/blob/main/README.md

Commands to clone the repository and install the required Python packages and libraries.

```shell
git clone 
cd MV2DFusion

pip install mmcls                     # 0.23.2
pip install mmcv-full                 # 1.6.1
pip install mmdet                     # 2.25.1
pip install mmdet3d                   # 1.0.0rc4
pip install mmsegmentation            # 0.28.0
pip install nuscenes-devkit
pip install av2
pip install refile                    # 0.4.1
pip install spconv-cu113              # 2.3.6
pip install flash-attn                # 1.0.2
pip install torch-scatter             # 2.1.2
pip install git+https://github.com/Abyssaledge/TorchEx.git
```

--------------------------------

### Launch Distributed Training

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Use these commands to initiate multi-GPU training for various configurations and datasets. Ensure the environment is configured for PyTorch distributed execution.

```bash
# Train MV2DFusion on nuScenes with 8 GPUs
bash tools/dist_train.sh projects/configs/nusc/mv2dfusion-fsd_freeze-r50_1600_gridmask-ep24_nusc.py 8

# Train with ConvNeXt-Large backbone for higher accuracy
bash tools/dist_train.sh projects/configs/nusc/mv2dfusion-fsd_freeze-convnextl_1600_gridmask-ep24_nusc.py 8

# Train on Argoverse2 dataset
bash tools/dist_train.sh projects/configs/argo/mv2dfusion-fsd_freeze-r50_1536-ep6_argov2.py 8

# Resume training from checkpoint
bash tools/dist_train.sh projects/configs/nusc/mv2dfusion-fsd_freeze-r50_1600_gridmask-ep24_nusc.py 8 --resume-from work_dirs/mv2dfusion/latest.pth

# Customize training parameters
PORT=29501 bash tools/dist_train.sh config.py 8 --cfg-options optimizer.lr=2e-4 data.samples_per_gpu=1
```

--------------------------------

### Initialize Custom nuScenes Dataset

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Configures the dataset loader for temporal sequences and demonstrates evaluation metrics.

```python
from mmdet3d.datasets import build_dataset
from mmcv import Config

# Dataset configuration
dataset_cfg = dict(
    type='CustomNuScenesDataset',
    data_root='./data/nuscenes/',
    ann_file='./data/nuscenes/nuscenes2d_temporal_infos_train.pkl',
    pipeline=train_pipeline,
    classes=['car', 'truck', 'construction_vehicle', 'bus', 'trailer',
             'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'],
    modality=dict(use_lidar=True, use_camera=True, use_radar=False, use_map=False),
    collect_keys=['lidar2img', 'intrinsics', 'extrinsics', 'timestamp',
                  'img_timestamp', 'ego_pose', 'ego_pose_inv', 'img', 'prev_exists', 'img_metas'],
    queue_length=1,                 # Number of historical frames
    num_frame_losses=1,             # Frames to compute loss
    seq_mode=True,                  # Enable streaming video training
    seq_split_num=2,                # Sequence split for training
    test_mode=False,
    use_valid_flag=True,
    filter_empty_gt=False,
    box_type_3d='LiDAR',
)

dataset = build_dataset(dataset_cfg)

# Access single sample
sample = dataset[0]
# sample keys: 'img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_bboxes',
#              'gt_labels', 'lidar2img', 'intrinsics', 'extrinsics', 'img_metas'

# Evaluation
results = [{'pts_bbox': {'boxes_3d': boxes, 'scores_3d': scores, 'labels_3d': labels}}]
metrics = dataset.evaluate(results, metric='bbox')
# Returns: {'pts_bbox_NuScenes/mAP': 0.73, 'pts_bbox_NuScenes/NDS': 0.75, ...}
```

--------------------------------

### Configure and Run Image Query Generator

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Initializes the ImageDistributionQueryGenerator with depth probability estimation and performs a forward pass using 2D detections and image metadata.

```python
from projects.mmdet3d_plugin.models.builder import build_query_generator

# Configuration for image query generator
img_query_generator_cfg = dict(
    type='ImageDistributionQueryGenerator',
    prob_bin=25,                    # Number of depth probability bins
    depth_range=[0.1, 90],          # Min/max depth range in meters
    gt_guided=False,                # Use GT depth supervision during training
    gt_guided_loss=1.0,             # Weight for depth supervision loss
    with_cls=True,                  # Predict classification scores
    with_size=True,                 # Predict 3D object sizes
    with_avg_pool=True,             # Use average pooling on RoI features
    num_shared_convs=1,             # Shared conv layers
    num_shared_fcs=1,               # Shared FC layers
    in_channels=256,                # Input feature channels
    fc_out_channels=1024,           # FC output channels
    roi_feat_size=7,                # RoI align output size
    extra_encoding=dict(
        num_layers=2,
        feat_channels=[512, 256],
        features=[dict(type='intrinsic', in_channels=16)]  # Encode camera intrinsics
    ),
)

img_query_generator = build_query_generator(img_query_generator_cfg)

# Forward pass
# x: RoI features [N_rois, C, H, W]
# proposal_list: List of 2D detections per image [x1, y1, x2, y2, score, class]
# img_metas: Image metadata with intrinsics/extrinsics
center_pred, return_feats = img_query_generator(
    x=roi_features,
    proposal_list=detections_2d,
    img_metas=img_metas,
    n_rois_per_view=n_rois_per_view,    # Number of RoIs per camera view
    n_rois_per_batch=n_rois_per_batch,  # Number of RoIs per batch
    data={'gt_bboxes': gt_bboxes, 'depths': depths}
)
# center_pred: Tuple of [N_batch, N_queries, prob_bin, 4] - 3D positions with depth probabilities
# return_feats: Dict with 'query_feats', 'depth_logits', 'cls_scores', 'bbox_preds'
```

--------------------------------

### Project Directory Structure

Source: https://github.com/wangzt-halo/mv2dfusion/blob/main/README.md

Expected file and folder layout after completing data preparation.

```text
MV2D
├── projects/
├── tools/
├── data/
│   ├── nuscenes/
│     ├── samples/
│     ├── sweeps/
│     ├── maps/
│     ├── v1.0-trainval/
│     ├── v1.0-test/
│     ├── nuscenes2d_temporal_infos_train.pkl
│     ├── nuscenes2d_temporal_infos_val.pkl
│     ├── nuscenes2d_temporal_infos_test.pkl
│     ├── ...
│   ├── argo/
│     ├── converted/
|         ├── av2_train_infos_mini.pkl
|         ├── av2_val_infos_mini.pkl
|         ├── val_anno.feather
|         ├── train -> ../sensor/train
|         ├── val -> ../sensor/val
|         ├── ...
│     ├── sensor/
|         ├── train/
|         ├── val/
|         ├── test/
|         ├── ...
├── weights/
├── README.md
```

--------------------------------

### Configure and Run Point Cloud Query Generator

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Initializes the PointCloudQueryGenerator for FSDv2 backbone outputs and executes the forward pass with voxelized LiDAR features.

```python
from projects.mmdet3d_plugin.models.builder import build_query_generator

# Configuration for point cloud query generator
pts_query_generator_cfg = dict(
    type='PointCloudQueryGenerator',
    in_channels=128,                # Input feature channels from FSDv2
    hidden_channel=128,             # Hidden layer channels
    pts_use_cat=False,              # Use class embedding
    dataset='nuscenes',             # Dataset type: 'nuscenes' or 'argov2'
    virtual_voxel_size=[0.4, 0.4, 0.4],
    point_cloud_range=[-54.4, -54.4, -5.0, 54.4, 54.4, 3.0],
    head_pc_range=[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0],
)

pts_query_generator = build_query_generator(pts_query_generator_cfg)

# Forward pass with FSDv2 backbone outputs
lidar_feat, lidar_pos, query_feat, query_pos = pts_query_generator(
    lidar_feat=voxel_feats,         # [N_voxels, C] voxel features
    lidar_indices=voxel_coors,      # [N_voxels, 4] voxel coordinates (batch, z, y, x)
    lidar_xyz=voxel_xyz,            # [N_voxels, 3] voxel center positions
    query_feat=query_feats,         # List of [N_queries, C] per batch
    query_xyz=query_xyz,            # List of [N_queries, 3] query positions
    query_pred=query_pred,          # List of [N_queries, 7] bbox predictions
    query_cat=query_cat,            # List of [N_queries] class predictions
    batch_size=batch_size
)
# lidar_feat: [B, max_voxels, C] padded BEV features
# lidar_pos: [B, max_voxels, 2] normalized BEV positions
# query_feat: [B, max_queries, C] padded query features
# query_pos: [B, max_queries, 3] query 3D positions
```

--------------------------------

### Configure and Initialize MV2DFusion Head

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Defines the architecture parameters for the fusion head and demonstrates the forward pass and bounding box extraction.

```python
fusion_head_cfg = dict(
    type='MV2DFusionHead',
    prob_bin=25,                    # Depth probability bins (must match img_query_generator)
    num_classes=10,                 # Number of object classes
    in_channels=256,                # Input embedding dimensions
    num_query=300,                  # Number of learnable queries
    memory_len=1536,                # Temporal memory length (frames * queries)
    topk_proposals=256,             # Top-k proposals for memory update
    num_propagated=256,             # Number of propagated queries
    with_ego_pos=True,              # Use ego-motion encoding
    match_with_velo=True,           # Match using velocity in Hungarian assignment
    scalar=10,                      # Denoising groups
    noise_scale=1.0,                # Denoising noise scale
    dn_weight=1.0,                  # Denoising loss weight
    split=0.75,                     # Positive rate for denoising
    code_weights=[2.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
    transformer=dict(
        type='MV2DFusionTransformer',
        decoder=dict(
            type='MV2DFusionTransformerDecoder',
            return_intermediate=True,
            num_layers=6,
            transformerlayers=dict(
                type='MV2DFusionTransformerDecoderLayer',
                attn_cfgs=[
                    dict(type='MultiheadAttention', embed_dims=256, num_heads=8, dropout=0.1),
                    dict(type='MixedCrossAttention', embed_dims=256, num_groups=8, num_levels=4,
                         num_cams=6, dropout=0.1, num_pts=13, bias=2.0),
                ],
                feedforward_channels=2048,
                ffn_dropout=0.1,
                with_cp=True,
            ),
        )
    ),
    bbox_coder=dict(
        type='NMSFreeCoder',
        post_center_range=[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0],
        pc_range=[-54.4, -54.4, -5.0, 54.4, 54.4, 3.0],
        max_num=300,
        voxel_size=[0.2, 0.2, 8],
        num_classes=10
    ),
    loss_cls=dict(type='FocalLoss', use_sigmoid=True, gamma=2.0, alpha=0.25, loss_weight=2.0),
    loss_bbox=dict(type='L1Loss', loss_weight=0.25),
)

fusion_head = build_head(fusion_head_cfg)

# Forward pass
outs = fusion_head(
    img_metas=img_metas,
    dyn_query=image_queries,           # From ImageDistributionQueryGenerator
    dyn_feats=image_query_feats,       # Query features dict
    pts_query_center=pts_query_pos,    # From PointCloudQueryGenerator
    pts_query_feat=pts_query_feat,     # Point cloud query features
    pts_feat=lidar_feat,               # BEV lidar features
    pts_pos=lidar_pos,                 # BEV positions
    prev_exists=prev_exists,           # Temporal context flag
    timestamp=timestamp,               # Frame timestamp
    ego_pose=ego_pose,                 # Ego vehicle pose
    lidar2img=lidar2img,               # Projection matrices
    img_feats_for_det=img_feats,       # Multi-scale image features
)
# Returns: {'all_cls_scores': [L, B, N, C], 'all_bbox_preds': [L, B, N, 10], 'dn_mask_dict': dict}

# Get final bounding boxes
bbox_results = fusion_head.get_bboxes(outs, img_metas)
# Returns: List of [LiDARInstance3DBoxes, scores, labels] per batch
```

--------------------------------

### Initialize and Run MV2DFusion Detector

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Demonstrates loading the model configuration and performing forward passes for training and inference. Requires MMDetection3D and MMCV dependencies.

```python
from mmdet.models import build_detector
from mmcv import Config

# Load configuration and build model
cfg = Config.fromfile('projects/configs/nusc/mv2dfusion-fsd_freeze-r50_1600_gridmask-ep24_nusc.py')
model = build_detector(cfg.model, train_cfg=cfg.train_cfg, test_cfg=cfg.test_cfg)

# Model architecture overview:
# - img_backbone: ResNet50 or ConvNeXt-Large for image feature extraction
# - img_neck: FPN for multi-scale image features
# - img_roi_head: 2D object detector (Faster R-CNN) for generating 2D proposals
# - img_query_generator: Converts 2D detections to 3D query distributions
# - pts_backbone: FSDv2 sparse voxel network for point cloud processing
# - pts_query_generator: Generates point cloud-based 3D queries
# - fusion_bbox_head: MV2DFusionHead transformer decoder for final 3D detection

# Forward pass for training
losses = model.forward_train(
    img_metas=img_metas,           # List of image metadata dicts
    gt_bboxes_3d=gt_bboxes_3d,     # Ground truth 3D boxes per batch
    gt_labels_3d=gt_labels_3d,     # Ground truth labels per batch
    gt_bboxes=gt_bboxes,           # 2D bounding boxes per view
    gt_labels=gt_labels,           # 2D labels per view
    depths=depths,                  # Depth annotations for 2D boxes
    centers2d=centers2d,           # 2D center projections
    img=images,                    # [B, T, N, C, H, W] multi-view images
    points=points,                 # LiDAR point clouds
    lidar2img=lidar2img,           # Transformation matrices
    intrinsics=intrinsics,         # Camera intrinsics
    extrinsics=extrinsics,         # Camera extrinsics
)
# Returns dict with losses: loss_cls, loss_bbox, pts.loss_*, det2d.*, imgqg.*

# Forward pass for inference
results = model.simple_test(img_metas, img=images, points=points, **data)
# Returns: [{'pts_bbox': {'boxes_3d': LiDARInstance3DBoxes, 'scores_3d': Tensor, 'labels_3d': Tensor}}]
```

--------------------------------

### Base Configuration Structure

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Defines the base configuration files and plugin directories for custom modules. Specifies training hyperparameters, point cloud settings, pretrained weights, and optimizer/learning rate configurations.

```python
_base_ = [
    '../_base_/datasets/nus-3d.py',
    '../_base_/default_runtime.py',
]

plugin = True
plugin_dir = ['projects/mmdet3d_plugin/', 'projects/fsdv2/']

num_gpus = 8
batch_size = 2
num_iters_per_epoch = 28130 // (num_gpus * batch_size)
num_epochs = 24

voxel_size = [0.2, 0.2, 8]
point_cloud_range = [-54.4, -54.4, -5.0, 54.4, 54.4, 3.0]

pts_ckpt = 'weights/fsdv2-converted.pth'
img_ckpt = 'weights/mask_rcnn_r50_fpn_1x_nuim_20201008_195238-e99f5182.pth'

optimizer = dict(
    type='AdamW',
    lr=4e-4,
    paramwise_cfg=dict(
        custom_keys={
            'img_backbone': dict(lr_mult=0.1),
            'pts_backbone': dict(lr_mult=0.1),
        }),
    weight_decay=0.01
)

lr_config = dict(
    policy='CosineAnnealing',
    warmup='linear',
    warmup_iters=500,
    warmup_ratio=1.0 / 3,
    min_lr_ratio=1e-3,
)

ida_aug_conf = {
    "resize_lim": (0.94, 1.25),
    "final_dim": (640, 1600),
    "rand_flip": True,
}

runner = dict(type='IterBasedRunner', max_iters=num_epochs * num_iters_per_epoch)
checkpoint_config = dict(interval=num_iters_per_epoch)
evaluation = dict(interval=3 * num_iters_per_epoch, pipeline=test_pipeline)
```

--------------------------------

### Initialize MV2DFusion Head

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Imports the builder for the transformer-based fusion head.

```python
from mmdet.models import build_head
```

--------------------------------

### Train and Evaluate Model

Source: https://github.com/wangzt-halo/mv2dfusion/blob/main/README.md

Commands for distributed training and evaluation of the MV2DFusion model.

```bash
bash tools/dist_train.sh projects/configs/nusc/mv2dfusion-fsd_freeze-r50_1600_gridmask-ep24_nusc.py 8 
```

```bash
bash tools/dist_test.sh projects/configs/nusc/mv2dfusion-fsd_freeze-r50_1600_gridmask-ep24_nusc.py work_dirs/mv2dfusion-fsd_freeze-r50_1600_gridmask-ep24_nusc/latest.pth 8 --eval bbox
```

--------------------------------

### Define Test Pipeline Configuration

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

This configuration defines the sequence of operations for testing, including loading points and images, normalization, and formatting for 3D object detection.

```python
test_pipeline = [
    dict(type='LoadPointsFromFile', coord_type='LIDAR', load_dim=5, use_dim=5),
    dict(type='LoadPointsFromMultiSweeps', sweeps_num=9, load_dim=5,
         use_dim=[0, 1, 2, 3, 4], pad_empty_sweeps=True, remove_close=True),
    dict(type='LoadMultiViewImageFromFiles', to_float32=True),
    dict(type='ResizeCropFlipRotImage', data_aug_conf=ida_aug_conf, training=False),
    dict(type='NormalizeMultiviewImage', mean=[123.675, 116.28, 103.53],
         std=[58.395, 57.12, 57.375], to_rgb=True),
    dict(type='PadMultiViewImage', size_divisor=32),
    dict(type='PointsRangeFilter', point_cloud_range=point_cloud_range),
    dict(type='NormalizePoints'),
    dict(type='MultiScaleFlipAug3D', img_scale=(1333, 800), pts_scale_ratio=1, flip=False,
         transforms=[
             dict(type='PETRFormatBundle3D', collect_keys=collect_keys, class_names=class_names),
             dict(type='Collect3D', keys=['points', 'img'])
         ])
]
```

--------------------------------

### Training Data Pipeline

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Defines the data loading and augmentation pipeline for training. Includes loading points from files and sweeps, multi-view images, annotations, and applying various transformations like rotation, scaling, flipping, and normalization.

```python
train_pipeline = [
    dict(type='LoadPointsFromFile', coord_type='LIDAR', load_dim=5, use_dim=5),
    dict(type='LoadPointsFromMultiSweeps', sweeps_num=9, load_dim=5,
         use_dim=[0, 1, 2, 3, 4], pad_empty_sweeps=True, remove_close=True),
    dict(type='LoadMultiViewImageFromFiles', to_float32=True),
    dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True,
         with_bbox=True, with_label=True, with_bbox_depth=True),
    dict(type='ResizeCropFlipRotImage', data_aug_conf=ida_aug_conf, training=True),
    dict(type='BEVGlobalRotScaleTrans',
         rot_range=[-1.57075, 1.57075],
         translation_std=[0, 0, 0],
         scale_ratio_range=[0.95, 1.05],
         training=True),
    dict(type='BEVRandomFlip3D'),
    dict(type='ObjectRangeFilter', point_cloud_range=point_cloud_range),
    dict(type='ObjectNameFilter', classes=class_names),
    dict(type='PointsRangeFilter', point_cloud_range=point_cloud_range),
    dict(type='NormalizePoints'),
    dict(type='PointShuffle'),
    dict(type='NormalizeMultiviewImage', mean=[123.675, 116.28, 103.53],
         std=[58.395, 57.12, 57.375], to_rgb=True),
    dict(type='PadMultiViewImage', size_divisor=32),
    dict(type='PETRFormatBundle3D', class_names=class_names, collect_keys=collect_keys),
    dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d', 'img',
                                  'gt_bboxes', 'gt_labels', 'centers2d', 'depths']),
]
```

--------------------------------

### Evaluate Trained Models

Source: https://context7.com/wangzt-halo/mv2dfusion/llms.txt

Execute distributed evaluation scripts to validate model performance on datasets. Results can be saved to files or formatted for submission.

```bash
# Evaluate on nuScenes validation set
bash tools/dist_test.sh projects/configs/nusc/mv2dfusion-fsd_freeze-r50_1600_gridmask-ep24_nusc.py \
    work_dirs/mv2dfusion-fsd_freeze-r50_1600_gridmask-ep24_nusc/latest.pth 8 --eval bbox

# Evaluate ConvNeXt model
bash tools/dist_test.sh projects/configs/nusc/mv2dfusion-fsd_freeze-convnextl_1600_gridmask-ep24_nusc.py \
    work_dirs/convnext/latest.pth 8 --eval bbox

# Save evaluation results to file
bash tools/dist_test.sh config.py checkpoint.pth 8 --eval bbox --out results.pkl

# Format results for test server submission
bash tools/dist_test.sh config.py checkpoint.pth 8 --format-only
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.