### Install Fiftyone for Visualization

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Install the Fiftyone library for advanced dataset visualization and exploration.

```shell
pip install fiftyone
```

--------------------------------

### Install ONNX and ONNX-Simplifier

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Install the necessary Python packages for exporting models to ONNX format.

```shell
pip install onnx onnxsim
```

--------------------------------

### Setup RT-DETRv4 Environment

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Create and activate a conda environment for RT-DETRv4 and install dependencies. Ensure Python 3.11.9 is used.

```shell
conda create -n rtv4 python=3.11.9
conda activate rtv4
pip install -r requirements.txt
```

--------------------------------

### Model FLOPs and Parameter Benchmark

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Benchmark the model's FLOPs, MACs, and parameters. Ensure you have installed the required dependencies.

```bash
pip install -r tools/benchmark/requirements.txt

python tools/benchmark/get_info.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml
# Output example:
# Model FLOPs:78.5G   MACs:39.2G   Params:20105000
```

--------------------------------

### Install Benchmark Requirements

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Install Python dependencies needed for benchmarking the model's performance.

```shell
pip install -r tools/benchmark/requirements.txt
```

--------------------------------

### Visualize Predictions with FiftyOne

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Installs FiftyOne and launches a browser-based visualization session with prediction overlays. Requires a configuration file and a trained model checkpoint.

```bash
pip install fiftyone

python tools/visualization/fiftyone_vis.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    -r outputs/rtv4_s/best_stg2.pth
# Opens a browser-based Fiftyone session with prediction overlays
```

--------------------------------

### Get Model FLOPs, MACs, and Parameters

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Calculate and display the FLOPs, MACs, and number of parameters for the RT-DETRv4 model using its configuration file.

```shell
python tools/benchmark/get_info.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml
```

--------------------------------

### YAMLConfig for Model and Training Setup

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Load configurations, override settings, and access lazily-instantiated model components. Supports hierarchical YAML files with inheritance.

```python
from engine.core import YAMLConfig

# Load config with optional overrides
cfg = YAMLConfig(
    'configs/rtv4/rtv4_hgnetv2_s_coco.yml',
    device='cuda:0',
    use_amp=True,
    output_dir='./outputs/my_run'
)

# Lazily-instantiated model components
model          = cfg.model           # RTv4 nn.Module
teacher_model  = cfg.teacher_model   # DINOv3TeacherModel (frozen)
postprocessor  = cfg.postprocessor   # PostProcessor
criterion      = cfg.criterion       # RTv4Criterion
optimizer      = cfg.optimizer       # AdamW with per-layer LR groups
lr_scheduler   = cfg.lr_scheduler    # MultiStepLR or FlatCosineLRScheduler
train_loader   = cfg.train_dataloader
val_loader     = cfg.val_dataloader
ema            = cfg.ema             # ModelEMA (if use_ema: True)
scaler         = cfg.scaler          # GradScaler (if use_amp: True)
evaluator      = cfg.evaluator       # CocoEvaluator

# Inspect the raw YAML dict
print(cfg.yaml_cfg.keys())
# dict_keys(['task', 'model', 'criterion', 'postprocessor', 'optimizer',
#            'lr_scheduler', 'train_dataloader', 'val_dataloader',
#            'teacher_model', 'HybridEncoder', 'RTv4Criterion', ...])
```

--------------------------------

### Install Inference Requirements

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Install Python dependencies required for running inference scripts.

```shell
pip install -r tools/inference/requirements.txt
```

--------------------------------

### ONNX Export Dependencies and Command

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Install necessary dependencies and export the model to ONNX format with automatic checking and simplification. The output is saved to `best_stg2.onnx`.

```bash
# Install dependencies
pip install onnx onnxsim

# Export with automatic check and simplification
python tools/deployment/export_onnx.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    -r outputs/rtv4_s/best_stg2.pth \
    --check \
    --simplify
# Produces: outputs/rtv4_s/best_stg2.onnx
```

--------------------------------

### Start Safe Training Script

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Execute a shell script designed to safely resume or manage the training process, potentially handling interruptions.

```shell
bash tools/reference/safe_training.sh
```

--------------------------------

### ONNX Runtime Inference

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Run inference using an ONNX Runtime session. This example demonstrates loading an ONNX model and performing inference on an image.

```python
# The exported model has these ONNX I/O specs:
# Inputs:
#   images:            float32 [N, 3, 640, 640]
#   orig_target_sizes: int64   [N, 2]   (image W, H)
# Outputs:
#   labels:            int64   [N, 300]
#   boxes:             float32 [N, 300, 4]   (x1y1x2y2, pixel coords)
#   scores:            float32 [N, 300]

import onnxruntime as ort
import numpy as np
from PIL import Image
import torchvision.transforms as T
import torch

sess = ort.InferenceSession('best_stg2.onnx', providers=['CUDAExecutionProvider'])

im = Image.open('cat.jpg').convert('RGB').resize((640, 640))
img_np = T.ToTensor()(im).unsqueeze(0).numpy()          # [1,3,640,640]
orig_size = np.array([[640, 640]], dtype=np.int64)       # [1,2]

labels, boxes, scores = sess.run(
    None,
    {'images': img_np, 'orig_target_sizes': orig_size}
)

keep = scores[0] > 0.4
print("Boxes:", boxes[0][keep])
print("Labels:", labels[0][keep])
print("Scores:", scores[0][keep])
```

--------------------------------

### RTv4 Main Model Configuration

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Example YAML configuration for the RTv4 model, including dataset, optimizer, and distillation settings. This config uses an include system for base settings.

```yaml
# configs/rtv4/rtv4_hgnetv2_s_coco.yml
__include__: [
  '../dfine/dfine_hgnetv2_s_coco.yml',  # backbone + decoder defaults
  '../base/rtv4.yml'                      # RTv4-specific losses and scheduler
]

output_dir: ./outputs/rtv4_hgnetv2_s_coco

# DINOv3 teacher for knowledge distillation
teacher_model:
  type: "DINOv3TeacherModel"
  dinov3_repo_path: dinov3/
  dinov3_weights_path: pretrain/dinov3_vitb16_pretrain_lvd1689m.pth
  patch_size: 16
  mean: [0.485, 0.456, 0.406]
  std: [0.229, 0.224, 0.225]

# Project student encoder features to teacher embedding dim (768 for ViT-B)
HybridEncoder:
  distill_teacher_dim: 768

# Distillation loss weight and adaptive controller
RTv4Criterion:
  weight_dict:
    loss_distill: 5
  distill_adaptive_params:
    enabled: True
    rho: 11      # target percentage of encoder gradient norm
    delta: 1     # tolerance band
    default_weight: 20

optimizer:
  type: AdamW
  params:
    - params: '^(?=.*backbone)(?!.*bn).*$'
      lr: 0.0002           # lower LR for backbone
    - params: '^(?=.*(?:norm|bn)).*$'
      weight_decay: 0.     # no weight decay for norm layers
  lr: 0.0004
  betas: [0.9, 0.999]
  weight_decay: 0.0001

epoches: 132
flat_epoch: 64    # epochs with flat (warm) LR before cosine decay
no_aug_epoch: 12  # disable heavy augmentation near end of training

train_dataloader:
  dataset:
    transforms:
      policy:
        epoch: [4, 64, 120]   # progressive augmentation schedule
  collate_fn:
    mixup_epochs: [4, 64]
    stop_epoch: 120           # stop mosaic/mixup at this epoch

```

--------------------------------

### Customize Input Size for RT-DETRv4

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Adjust dataloader and base configuration files to set a custom input size for RT-DETRv4 training and evaluation. This example sets the input size to 320x320.

```yaml
train_dataloader:
  dataset:
      transforms:
          ops:
              - {type: Resize, size: [320, 320], }
  collate_fn:
      base_size: 320

val_dataloader:
  dataset:
      transforms:
          ops:
              - {type: Resize, size: [320, 320], }
```

```yaml
eval_spatial_size: [320, 320]
```

--------------------------------

### Customize Batch Size for RT-DETRv4

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Modify dataloader and model configuration files to adjust the total batch size. This example doubles the batch size for RT-DETRv4-L on COCO2017.

```yaml
train_dataloader:
    total_batch_size: 64  # Previously it was 32, now doubled
```

```yaml
optimizer:
  type: AdamW
  params:
    - 
      params: "^(?=.*backbone)(?!.*norm|bn).*$"
      lr: 0.000025  # doubled, linear scaling law
    - 
      params: "^(?=.*(?:encoder|decoder))(?=.*(?:norm|bn)).*$"
      weight_decay: 0.

lr: 0.0005  # doubled, linear scaling law
betas: [0.9, 0.999]
weight_decay: 0.0001  # need a grid search

ema:  # added EMA settings
    decay: 0.9998  # adjusted by 1 - (1 - decay) * 2
    warmups: 500  # halved

lr_warmup_scheduler:
    warmup_duration: 250  # halved
```

--------------------------------

### Visualize with Fiftyone

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Launch the Fiftyone visualization tool for RT-DETRv4. This command uses the model configuration and weights to display results.

```shell
python tools/visualization/fiftyone_vis.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth
```

--------------------------------

### Initialize DINOv3 Teacher Model

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Instantiate the DINOv3TeacherModel with specified repository, weights, and model type. Ensure the model is moved to CUDA and kept in eval mode with no_grad for inference.

```python
from engine.rtv4.dinov3_teacher import DINOv3TeacherModel
import torch

teacher = DINOv3TeacherModel(
    dinov3_repo_path='dinov3/',
    dinov3_weights_path='pretrain/dinov3_vitb16_pretrain_lvd1689m.pth',
    dinov3_model_type='dinov3_vitb16',
    patch_size=16,
    mean=(0.485, 0.456, 0.406),
    std=(0.229, 0.224, 0.225),
).cuda()

# Teacher is always kept in eval mode with no_grad
images = torch.rand(2, 3, 640, 640).cuda()
with torch.no_grad():
    teacher_features = teacher(images)
# teacher_features: [2, 768, 20, 20]  (patch_size=16, 640/2/16=20)
# All parameters are frozen (requires_grad=False)
print(teacher.teacher_feature_dim)  # 768 for ViT-B/16
```

--------------------------------

### PostProcessor Initialization and Usage

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Initializes and uses the PostProcessor to convert raw model outputs into detection results, including decoding bounding boxes and filtering by score.

```APIDOC
## PostProcessor

### Description
`engine.rtv4.postprocessor.PostProcessor` converts raw model outputs (`pred_logits`, `pred_boxes`) into per-image detection lists with rescaled bounding boxes. In deploy mode it returns flat tensors suitable for ONNX export.

### Initialization
```python
from engine.rtv4.postprocessor import PostProcessor
import torch

postprocessor = PostProcessor(
    num_classes=80,
    use_focal_loss=True,
    num_top_queries=300,
    remap_mscoco_category=False,
)
```

### Usage
```python
outputs = {
    'pred_logits': torch.rand(2, 300, 80),  # raw logits
    'pred_boxes':  torch.rand(2, 300, 4),   # cxcywh normalized
}
orig_target_sizes = torch.tensor([[1080, 1920], [720, 1280]])  # [B, 2] (H, W)

results = postprocessor(outputs, orig_target_sizes)
# results: list of dicts, one per image
# results[0] = {
#     'labels': tensor([17, 0, ...]),         # class ids, shape [300]
#     'boxes':  tensor([[x1,y1,x2,y2], ...]), # abs pixel coords, shape [300, 4]
#     'scores': tensor([0.91, 0.87, ...]),     # confidence, shape [300]
# }

# Filter by score threshold
r = results[0]
keep = r['scores'] > 0.5
print(r['boxes'][keep])   # high-confidence boxes only
print(r['labels'][keep])  # their class IDs
```

### Parameters
- `num_classes` (int): Number of object classes.
- `use_focal_loss` (bool): Whether focal loss was used during training.
- `num_top_queries` (int): The number of top queries to consider.
- `remap_mscoco_category` (bool): Whether to remap COCO categories.

### Returns
- `results` (list): A list of dictionaries, where each dictionary contains 'labels', 'boxes', and 'scores' for detections in an image.
```

--------------------------------

### Configure DINOv3 Teacher Model

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Specify the local paths to your DINOv3 repository and checkpoint file in the teacher model configuration.

```yaml
teacher_model:
  type: "DINOv3TeacherModel"
  dinov3_repo_path: dinov3/
  dinov3_weights_path: pretrain/dinov3_vitb16_pretrain_lvd1689m.pth
```

--------------------------------

### Run PyTorch Inference

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Perform inference using the original PyTorch model. Requires configuration file, model weights, and specified device.

```shell
python tools/inference/torch_inf.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth --input image.jpg --device cuda:0
```

--------------------------------

### Initialize RTv4 Criterion for Multi-Task Loss

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Set up the RTv4Criterion with a HungarianMatcher and a weight dictionary for various loss components. Configure loss types, class number, and distillation parameters.

```python
from engine.rtv4.rtv4_criterion import RTv4Criterion
from engine.rtv4.matcher import HungarianMatcher
import torch

matcher = HungarianMatcher(
    weight_dict={'cost_class': 2, 'cost_bbox': 5, 'cost_giou': 2},
    use_focal_loss=True,
    alpha=0.25,
    gamma=2.0,
)

criterion = RTv4Criterion(
    matcher=matcher,
    weight_dict={
        'loss_mal':     1.0,
        'loss_bbox':    5.0,
        'loss_giou':    2.0,
        'loss_fgl':     0.15,
        'loss_ddf':     1.5,
        'loss_distill': 10.0,
    },
    losses=['mal', 'boxes', 'local', 'distill'],
    alpha=0.2,
    gamma=1.5,
    num_classes=80,
    reg_max=32,
    use_uni_set=True,
    distill_adaptive_params={'enabled': True, 'rho': 11, 'delta': 1, 'default_weight': 20},
).cuda()

# outputs and targets come from RTv4.forward()
losses = criterion(outputs, targets)
# losses dict example:
# {'loss_mal': tensor(0.42), 'loss_bbox': tensor(1.12), 'loss_giou': tensor(0.67),
#  'loss_fgl': tensor(0.05), 'loss_ddf': tensor(0.31), 'loss_distill': tensor(0.18),
#  'loss_mal_aux_0': ..., 'loss_bbox_dn_0': ..., ...}

total_loss = sum(losses.values())
total_loss.backward()
```

--------------------------------

### RTv4 Model Training and Inference

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Python code demonstrating how to instantiate the RTv4 model from a YAML config, perform training forward passes with targets and teacher output, and inference forward passes. Includes model deployment for fast inference.

```python
import torch
from engine.core import YAMLConfig

cfg = YAMLConfig('configs/rtv4/rtv4_hgnetv2_s_coco.yml')
model = cfg.model   # RTv4 instance

# ----- Training forward pass ----- 
images = torch.rand(2, 3, 640, 640, device='cuda')
targets = [
    {'labels': torch.tensor([0, 1]), 'boxes': torch.tensor([[0.5, 0.5, 0.2, 0.3],
                                                             [0.3, 0.4, 0.1, 0.2]])},
    {'labels': torch.tensor([2]),    'boxes': torch.tensor([[0.7, 0.7, 0.4, 0.4]])},
]

model.train().cuda()
teacher_output = cfg.teacher_model(images)   # [B, 768, H_t, W_t]

outputs = model(images, targets=targets, teacher_encoder_output=teacher_output)
# outputs keys (training): pred_logits, pred_boxes, pred_corners, ref_points,
#                           aux_outputs, dn_outputs, enc_aux_outputs,
#                           student_distill_output, teacher_encoder_output

# ----- Inference forward pass ----- 
model.eval().cuda()
with torch.no_grad():
    outputs = model(images)
# outputs keys (eval): pred_logits [B,300,80], pred_boxes [B,300,4]

# ----- Deploy mode (fuses BN into Conv for fast inference) ----- 
deployed_model = model.deploy()

```

--------------------------------

### Initialize PostProcessor for Decoding Detections

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Instantiate the PostProcessor to convert raw model outputs into detection results. Configure class mapping and query handling. The processor can output flat tensors for ONNX export.

```python
from engine.rtv4.postprocessor import PostProcessor
import torch

postprocessor = PostProcessor(
    num_classes=80,
    use_focal_loss=True,
    num_top_queries=300,
    remap_mscoco_category=False,
)

outputs = {
    'pred_logits': torch.rand(2, 300, 80),  # raw logits
    'pred_boxes':  torch.rand(2, 300, 4),   # cxcywh normalized
}
orig_target_sizes = torch.tensor([[1080, 1920], [720, 1280]])  # [B, 2] (H, W)

results = postprocessor(outputs, orig_target_sizes)
# results: list of dicts, one per image
# results[0] = {
#     'labels': tensor([17, 0, ...]),         # class ids, shape [300]
#     'boxes':  tensor([[x1,y1,x2,y2], ...]), # abs pixel coords, shape [300, 4]
#     'scores': tensor([0.91, 0.87, ...]),     # confidence, shape [300]
# }

# Filter by score threshold
r = results[0]
keep = r['scores'] > 0.5
print(r['boxes'][keep])   # high-confidence boxes only
print(r['labels'][keep])  # their class IDs
```

--------------------------------

### Train on Custom Dataset with RT-DETRv4-S

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Command to train RT-DETRv4-S on a custom dataset, overriding default configurations for dataset paths and number of classes. Uses distributed training with AMP.

```bash
# Train on custom dataset using RT-DETRv4-S
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 \
    train.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    --use-amp --seed=0 \
    -u num_classes=10 \
       train_dataloader.dataset.img_folder=/data/mydata/train/images \
       train_dataloader.dataset.ann_file=/data/mydata/train/annotations.json \
       val_dataloader.dataset.img_folder=/data/mydata/val/images \
       val_dataloader.dataset.ann_file=/data/mydata/val/annotations.json
```

--------------------------------

### RT-DETRv4 Training and Evaluation Commands

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Execute training, multi-GPU training, resume training, fine-tuning, or evaluation using the train.py script. Override YAML configurations directly from the command line.

```bash
python train.py -c configs/rtv4/rtv4_hgnetv2_s_coco.yml --use-amp --seed=0
```

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 \
    train.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    --use-amp \
    --seed=0 \
    --output-dir ./outputs/rtv4_s
```

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 \
    train.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    --use-amp --seed=0 \
    -r outputs/rtv4_s/last.pth
```

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 \
    train.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    --use-amp --seed=0 \
    -t pretrained/rtv4_s_obj365.pth
```

```bash
CUDA_VISIBLE_DEVICES=0 python train.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    --test-only \
    -r outputs/rtv4_s/best_stg2.pth
```

```bash
python train.py -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    -u epoches=50 optimizer.lr=0.0002
```

--------------------------------

### Interactive Safe Training Script

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Launches an interactive shell script for training that allows selection of model size and task. It includes automatic restart on failure using the last saved checkpoint.

```bash
bash tools/reference/safe_training.sh
# Interactive prompts:
#   Select model size: s / m / l / x
#   Select task: obj365 / obj2coco / coco
#   Save logs to txt? y/n
# Runs: torchrun ... train.py -c configs/dfine/dfine_hgnetv2_s_coco.yml ...
# On crash, auto-resumes from last.pth
```

--------------------------------

### Initialize Hungarian Matcher for Bipartite Assignment

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Create a HungarianMatcher instance to solve the assignment problem between predictions and ground truths. Configure costs for classification, bounding box, and GIoU, with options for focal loss.

```python
from engine.rtv4.matcher import HungarianMatcher
import torch

matcher = HungarianMatcher(
    weight_dict={'cost_class': 2, 'cost_bbox': 5, 'cost_giou': 2},
    use_focal_loss=True,
    alpha=0.25,
    gamma=2.0,
)

outputs = {
    'pred_logits': torch.rand(2, 300, 80),   # [B, num_queries, num_classes]
    'pred_boxes':  torch.rand(2, 300, 4),    # [B, num_queries, 4] cxcywh in [0,1]
}
targets = [
    {'labels': torch.tensor([0, 5]),  'boxes': torch.rand(2, 4)},
    {'labels': torch.tensor([12]),    'boxes': torch.rand(1, 4)},
]

result = matcher(outputs, targets)
indices = result['indices']
# indices: list of 2 tuples (pred_idx, gt_idx) — one per batch element
# e.g. [(tensor([14, 207]), tensor([0, 1])),
#        (tensor([88]),      tensor([0]))]
```

--------------------------------

### Run ONNX Runtime Inference

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Perform inference on images or videos using an ONNX model with ONNX Runtime.

```shell
python tools/inference/onnx_inf.py --onnx model.onnx --input image.jpg  # or video.mp4
```

--------------------------------

### Train Prior Works with RT-DETRv4 Codebase

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Commands to train prior works like DEIM, D-FINE, and RT-DETRv2 using the RT-DETRv4 codebase with specified configurations and AMP enabled.

```bash
# DEIM (with HGNetv2 backbone)
python train.py -c configs/deim/deim_hgnetv2_s_coco.yml --use-amp

# D-FINE
python train.py -c configs/dfine/dfine_hgnetv2_s_coco.yml --use-amp

# RT-DETRv2 (ResNet backbone)
python train.py -c configs/rtv2/rtv2_r50vd_6x_coco.yml --use-amp
```

--------------------------------

### Base RTv4 Loss and Scheduler Settings

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

YAML configuration for base RTv4 settings, defining loss weights, scheduler type, and augmentation schedules. This file is included by other model configurations.

```yaml
# configs/base/rtv4.yml — shared loss and scheduler settings
RTv4Criterion:
  weight_dict: {loss_mal: 1, loss_bbox: 5, loss_giou: 2, loss_fgl: 0.15, loss_ddf: 1.5, loss_distill: 10.0}
  losses: ['mal', 'boxes', 'local', 'distill']
  gamma: 1.5

lrsheduler: flatcosine
lr_gamma: 0.5
warmup_iter: 2000
flat_epoch: 29
no_aug_epoch: 8

HGNetv2:
  freeze_at: -1       # do not freeze any backbone stage
  freeze_norm: False  # allow BN stats to update

DFINETransformer:
  activation: silu
  mlp_act: silu

```

--------------------------------

### RTv4Criterion Initialization and Usage

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Initializes and uses the RTv4Criterion to compute the combined training loss, supporting various loss types including classification, regression, and distillation.

```APIDOC
## RTv4Criterion

### Description
`engine.rtv4.rtv4_criterion.RTv4Criterion` computes the combined training loss. It supports focal/VFL/MAL classification losses, L1+GIoU box regression, Fine-Grained Localization (FGL), Decoupled Distillation Focal (DDF), and the new VFM cosine-similarity distillation loss across all decoder layers.

### Initialization
```python
from engine.rtv4.rtv4_criterion import RTv4Criterion
from engine.rtv4.matcher import HungarianMatcher
import torch

matcher = HungarianMatcher(
    weight_dict={'cost_class': 2, 'cost_bbox': 5, 'cost_giou': 2},
    use_focal_loss=True,
    alpha=0.25,
    gamma=2.0,
)

criterion = RTv4Criterion(
    matcher=matcher,
    weight_dict={
        'loss_mal':     1.0,
        'loss_bbox':    5.0,
        'loss_giou':    2.0,
        'loss_fgl':     0.15,
        'loss_ddf':     1.5,
        'loss_distill': 10.0,
    },
    losses=['mal', 'boxes', 'local', 'distill'],
    alpha=0.2,
    gamma=1.5,
    num_classes=80,
    reg_max=32,
    use_uni_set=True,
    distill_adaptive_params={'enabled': True, 'rho': 11, 'delta': 1, 'default_weight': 20},
).cuda()
```

### Usage
```python
# outputs and targets come from RTv4.forward()
losses = criterion(outputs, targets)
# losses dict example:
# {'loss_mal': tensor(0.42), 'loss_bbox': tensor(1.12), 'loss_giou': tensor(0.67),
#  'loss_fgl': tensor(0.05), 'loss_ddf': tensor(0.31), 'loss_distill': tensor(0.18),
#  'loss_mal_aux_0': ..., 'loss_bbox_dn_0': ..., ...}

total_loss = sum(losses.values())
total_loss.backward()
```

### Parameters
- `matcher` (HungarianMatcher): An instance of HungarianMatcher for bipartite assignment.
- `weight_dict` (dict): Dictionary mapping loss names to their weights.
- `losses` (list): List of loss types to compute (e.g., ['mal', 'boxes', 'local', 'distill']).
- `alpha` (float): Alpha parameter for focal loss.
- `gamma` (float): Gamma parameter for focal loss.
- `num_classes` (int): Number of object classes.
- `reg_max` (int): Maximum value for regression.
- `use_uni_set` (bool): Whether to use unified set prediction.
- `distill_adaptive_params` (dict): Parameters for adaptive distillation.

### Returns
- `losses` (dict): A dictionary containing the computed loss values.
```

--------------------------------

### Configure Custom Detection Dataset

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

YAML configuration for a custom detection dataset. Update `num_classes` and dataset paths to match your specific dataset.

```yaml
task: detection

evaluator:
  type: CocoEvaluator
  iou_types: ['bbox', ]

num_classes: 777 # your dataset classes
remap_mscoco_category: False

train_dataloader:
  type: DataLoader
  dataset:
    type: CocoDetection
    img_folder: /data/yourdataset/train
    ann_file: /data/yourdataset/train/train.json
    return_masks: False
    transforms:
      type: Compose
      ops: ~
  shuffle: True
  num_workers: 4
  drop_last: True
  collate_fn:
    type: BatchImageCollateFunction

val_dataloader:
  type: DataLoader
  dataset:
    type: CocoDetection
    img_folder: /data/yourdataset/val
    ann_file: /data/yourdataset/val/ann.json
    return_masks: False
    transforms:
      type: Compose
      ops: ~
  shuffle: False
  num_workers: 4
  drop_last: False
  collate_fn:
    type: BatchImageCollateFunction
```

--------------------------------

### HungarianMatcher Initialization and Usage

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Initializes and uses the HungarianMatcher to solve the linear sum assignment problem for matching predictions to ground-truth boxes.

```APIDOC
## HungarianMatcher

### Description
`engine.rtv4.matcher.HungarianMatcher` solves the linear sum assignment problem to match predictions to ground-truth boxes using a combined cost of focal classification, L1 box regression, and GIoU.

### Initialization
```python
from engine.rtv4.matcher import HungarianMatcher
import torch

matcher = HungarianMatcher(
    weight_dict={'cost_class': 2, 'cost_bbox': 5, 'cost_giou': 2},
    use_focal_loss=True,
    alpha=0.25,
    gamma=2.0,
)
```

### Usage
```python
outputs = {
    'pred_logits': torch.rand(2, 300, 80),   # [B, num_queries, num_classes]
    'pred_boxes':  torch.rand(2, 300, 4),    # [B, num_queries, 4] cxcywh in [0,1]
}
targets = [
    {'labels': torch.tensor([0, 5]),  'boxes': torch.rand(2, 4)},
    {'labels': torch.tensor([12]),    'boxes': torch.rand(1, 4)},
]

result = matcher(outputs, targets)
indices = result['indices']
# indices: list of 2 tuples (pred_idx, gt_idx) — one per batch element
# e.g. [(tensor([14, 207]), tensor([0, 1])),
#        (tensor([88]),      tensor([0]))]
```

### Parameters
- `weight_dict` (dict): Dictionary specifying the weights for classification, bounding box, and GIoU costs.
- `use_focal_loss` (bool): Whether to use focal loss for classification cost.
- `alpha` (float): Alpha parameter for focal loss.
- `gamma` (float): Gamma parameter for focal loss.

### Returns
- `result` (dict): A dictionary containing the `indices` for matching.
```

--------------------------------

### PyTorch Image Inference

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Run inference on an image file using a trained PyTorch model. Specify the configuration, checkpoint, input image, and target device.

```bash
python tools/inference/torch_inf.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    -r outputs/rtv4_s/best_stg2.pth \
    -i path/to/image.jpg \
    -d cuda:0
# Saves annotated result to torch_results.jpg
```

--------------------------------

### DINOv3TeacherModel Initialization and Usage

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Initializes and uses the DINOv3TeacherModel for feature extraction. This model wraps a pre-trained DINOv3 ViT model with frozen parameters.

```APIDOC
## DINOv3TeacherModel

### Description
`engine.rtv4.dinov3_teacher.DINOv3TeacherModel` wraps a locally-loaded DINOv3 ViT model with frozen parameters. It applies 2×2 average pooling to the input image before passing it to the teacher, then reshapes patch tokens into a spatial feature map.

### Initialization
```python
from engine.rtv4.dinov3_teacher import DINOv3TeacherModel
import torch

teacher = DINOv3TeacherModel(
    dinov3_repo_path='dinov3/',
    dinov3_weights_path='pretrain/dinov3_vitb16_pretrain_lvd1689m.pth',
    dinov3_model_type='dinov3_vitb16',
    patch_size=16,
    mean=(0.485, 0.456, 0.406),
    std=(0.229, 0.224, 0.225),
).cuda()
```

### Usage
```python
# Teacher is always kept in eval mode with no_grad
images = torch.rand(2, 3, 640, 640).cuda()
with torch.no_grad():
    teacher_features = teacher(images)
# teacher_features: [2, 768, 20, 20]  (patch_size=16, 640/2/16=20)
```

### Teacher Feature Dimension
```python
print(teacher.teacher_feature_dim)  # 768 for ViT-B/16
```

### Parameters
- `dinov3_repo_path` (str): Path to the DINOv3 repository.
- `dinov3_weights_path` (str): Path to the DINOv3 weights file.
- `dinov3_model_type` (str): Type of the DINOv3 model (e.g., 'dinov3_vitb16').
- `patch_size` (int): The patch size of the ViT model.
- `mean` (tuple): Mean values for image normalization.
- `std` (tuple): Standard deviation values for image normalization.

### Returns
- `teacher_features` (torch.Tensor): The extracted features from the teacher model.
```

--------------------------------

### Train RT-DETRv4 on COCO2017

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Use this command to initiate training for RT-DETRv4 on the COCO2017 dataset. Ensure you have the necessary configuration files and model weights.

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml --use-amp --seed=0
```

--------------------------------

### Custom Dataset Directory Structure

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Organize your custom dataset images and annotations in a structure compatible with the COCO format.

```shell
dataset/
    ├── images/
    │   ├── train/
    │   │   ├── image1.jpg
    │   │   ├── image2.jpg
    │   │   └── ...
    │   ├── val/
    │   │   ├── image1.jpg
    │   │   ├── image2.jpg
    │   │   └── ...
    └── annotations/
        ├── instances_train.json
        ├── instances_val.json
        └── ...
```

--------------------------------

### Convert Model Weights

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Convert model weights from one format to another. This is useful for compatibility or migration purposes.

```shell
python tools/reference/convert_weight.py model.pth
```

--------------------------------

### HybridEncoder Initialization and Usage

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Python code for initializing the HybridEncoder, which processes backbone features through transformers and FPN for detection. It supports projecting student features for distillation when `distill_teacher_dim` is set.

```python
from engine.rtv4.hybrid_encoder import HybridEncoder
import torch

encoder = HybridEncoder(
    in_channels=[512, 1024, 2048],   # HGNetv2-S output channels
    feat_strides=[8, 16, 32],
    hidden_dim=256,
    nhead=8,
    dim_feedforward=1024,
    dropout=0.0,
    enc_act='gelu',
    use_encoder_idx=[2],             # apply transformer on S5 only
    num_encoder_layers=1,
    expansion=1.0,
    depth_mult=1.0,
    act='silu',
    eval_spatial_size=[640, 640],
    version='dfine',                 # use RepNCSPELAN4 blocks
    distill_teacher_dim=768,         # project to DINOv3 ViT-B dim
).cuda()

# Backbone features (S3, S4, S5)
feats = [
    torch.rand(2, 512, 80, 80).cuda(),
    torch.rand(2, 1024, 40, 40).cuda(),
    torch.rand(2, 2048, 20, 20).cuda(),
]

encoder.train()
result = encoder(feats)
fpn_features, student_distill_out = result   # training returns tuple

```

--------------------------------

### PyTorch Video Inference

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Perform inference on a video file using a trained PyTorch model. Similar to image inference, specify configuration, checkpoint, input video, and device.

```bash
python tools/inference/torch_inf.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    -r outputs/rtv4_s/best_stg2.pth \
    -i path/to/video.mp4 \
    -d cuda:0
# Saves annotated video to torch_results.mp4
```

--------------------------------

### Export Model to ONNX Format

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Use this command to export the trained RT-DETRv4 model to the ONNX format. The --check flag performs a verification after export.

```shell
python tools/deployment/export_onnx.py --check -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth
```

--------------------------------

### Run TensorRT Inference

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Perform inference using a TensorRT engine. Specify the path to the engine file and the input image.

```shell
python tools/inference/trt_inf.py --trt model.engine --input image.jpg
```

--------------------------------

### ONNX Video Inference Command

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Run inference on a video using the exported ONNX model. The result is saved as `onnx_result.mp4`.

```bash
# Video
python tools/inference/onnx_inf.py \
    --onnx best_stg2.onnx \
    --input driving.mp4
# Saves: onnx_result.mp4
```

--------------------------------

### TensorRT Export and Inference Pipeline

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

A three-step process to export a model to ONNX, build a TensorRT engine (with FP16 optimization), and then run inference using the TensorRT engine.

```bash
# 1. Export to ONNX first
python tools/deployment/export_onnx.py \
    -c configs/rtv4/rtv4_hgnetv2_s_coco.yml \
    -r best_stg2.pth

# 2. Build TensorRT engine with FP16
trtexec \
    --onnx=best_stg2.onnx \
    --saveEngine=best_stg2.engine \
    --fp16

# 3. Run TRT inference
python tools/inference/trt_inf.py \
    --trt best_stg2.engine \
    --input image.jpg
# Saves: trt_result.jpg
```

--------------------------------

### Programmatic PyTorch Inference

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Use PyTorch programmatically to load a model, perform inference, and process results. This matches the internal logic of `torch_inf.py`.

```python
# Programmatic usage matching torch_inf.py internals
import torch
import torchvision.transforms as T
from PIL import Image
from engine.core import YAMLConfig
import torch.nn as nn

cfg = YAMLConfig('configs/rtv4/rtv4_hgnetv2_s_coco.yml', resume=None)
cfg.yaml_cfg['HGNetv2']['pretrained'] = False

checkpoint = torch.load('best_stg2.pth', map_location='cpu')
state = checkpoint['ema']['module'] if 'ema' in checkpoint else checkpoint['model']
cfg.model.load_state_dict(state)

class DeployModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = cfg.model.deploy()           # fuse BN→Conv
        self.postprocessor = cfg.postprocessor.deploy()
    def forward(self, images, orig_target_sizes):
        return self.postprocessor(self.model(images), orig_target_sizes)

device = 'cuda:0'
model = DeployModel().to(device).eval()

im = Image.open('dog.jpg').convert('RGB')
w, h = im.size
orig_size = torch.tensor([[w, h]]).to(device)
transform = T.Compose([T.Resize((640, 640)), T.ToTensor()])
img_tensor = transform(im).unsqueeze(0).to(device)

with torch.no_grad():
    labels, boxes, scores = model(img_tensor, orig_size)

keep = scores[0] > 0.4
print(f"Detected {keep.sum()} objects")
print("Labels:", labels[0][keep].tolist())
print("Scores:", scores[0][keep].tolist())
```

--------------------------------

### Test RT-DETRv4 on COCO2017

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Command to run testing and evaluation for RT-DETRv4 on COCO2017. Specify the trained model weights using the -r flag.

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml --test-only -r model.pth
```

--------------------------------

### Configure COCO2017 DataLoader

Source: https://github.com/rt-detrs/rt-detrv4/blob/main/README.md

Specify paths for COCO2017 training and validation data. Ensure the paths in this YAML file point to your local COCO2017 dataset directories.

```yaml
train_dataloader:
    img_folder: /data/COCO2017/train2017/
    ann_file: /data/COCO2017/annotations/instances_train2017.json
val_dataloader:
    img_folder: /data/COCO2017/val2017/
    ann_file: /data/COCO2017/annotations/instances_val2017.json
```

--------------------------------

### Custom Dataset Configuration YAML

Source: https://context7.com/rt-detrs/rt-detrv4/llms.txt

Defines a custom dataset configuration for object detection using the COCO format. Specifies dataset paths, number of classes, and dataloader settings.

```yaml
# configs/dataset/custom_detection.yml
task: detection

evaluator:
  type: CocoEvaluator
  iou_types: ['bbox']

num_classes: 10           # your number of classes
remap_mscoco_category: False

train_dataloader:
  type: DataLoader
  dataset:
    type: CocoDetection
    img_folder: /data/mydata/train/images
    ann_file: /data/mydata/train/annotations.json
    return_masks: False
    transforms:
      type: Compose
      ops: ~
  shuffle: True
  num_workers: 4
  drop_last: True
  total_batch_size: 32
  collate_fn:
    type: BatchImageCollateFunction

val_dataloader:
  type: DataLoader
  dataset:
    type: CocoDetection
    img_folder: /data/mydata/val/images
    ann_file: /data/mydata/val/annotations.json
    return_masks: False
    transforms:
      type: Compose
      ops: ~
  shuffle: False
  num_workers: 4
  drop_last: False
  total_batch_size: 8
  collate_fn:
    type: BatchImageCollateFunction
```