### Setup EfficientViT Environment

Source: https://github.com/mit-han-lab/efficientvit/blob/master/README.md

Use this command to create a new conda environment for EfficientViT and install dependencies. Ensure Python 3.10 is used.

```bash
conda create -n efficientvit python=3.10
conda activate efficientvit
pip install -U -r requirements.txt
```

--------------------------------

### Install Dependencies

Source: https://github.com/mit-han-lab/efficientvit/blob/master/assets/train_diffusion_with_flux_vae_f8.md

Install the 'accelerate' and 'sentencepiece' libraries required for the diffusion model training process.

```bash
pip install accelerate
pip install sentencepiece
```

--------------------------------

### Setup PyTorch Depth Estimation Model

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_gazesam/README.md

Copies a replacement DPT script into the Depth-Anything directory to prepend the torchhub local download path. Ensure the Depth-Anything repository is cloned and the checkpoint is downloaded.

```bash
cp models/create_pytorch/dpt_replacement.py Depth-Anything/depth_anything/dpt.py
```

--------------------------------

### Install Dependencies for EfficientViT-GazeSAM

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Installs the necessary extra dependencies for running the EfficientViT-GazeSAM pipeline. This command should be run before executing the GazeSAM demo.

```bash
# Install extra dependencies
pip install -r applications/efficientvit_gazesam/extra_requirements.txt
```

--------------------------------

### Install Extra Packages

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_gazesam/README.md

Installs additional required packages for the project. Ensure you have the `extra_requirements.txt` file available.

```bash
pip install -r extra_requirements.txt
```

--------------------------------

### Train EfficientViT-SAM Model

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Initiates training or fine-tuning of the EfficientViT-SAM model on the SA-1B dataset, starting from a distilled checkpoint. Requires a configuration file and specifies output paths.

```bash
# Place distilled checkpoints under assets/checkpoints/efficientvit_sam/distilled_model/
torchrun --nproc_per_node=8 \
    applications/efficientvit_sam/train_efficientvit_sam_model.py \
    applications/efficientvit_sam/configs/efficientvit_sam_xl1.yaml \
    --data_provider.root ~/dataset/sam/ \
    --path .exp/efficientvit_sam/efficientvit_sam_xl1 \
    --resume
```

--------------------------------

### Install ONNX Runtime GPU

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_gazesam/README.md

Installs the ONNX Runtime GPU package for ONNX model inference. If issues arise, try uninstalling and reinstalling `onnxruntime-gpu`.

```bash
python -m pip install onnxruntime-gpu
```

--------------------------------

### Install Diffusers Library

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Install the latest version of the diffusers library to use the AutoencoderDC model.

```bash
pip install -U diffusers
```

--------------------------------

### Train DC-AE-f32 + DiT-XL on ImageNet 512x512 (Batch Size 1024)

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

This command trains a diffusion model using DC-AE-f32 and DiT-XL on ImageNet 512x512 with a batch size of 1024. It specifies different learning rate and batch size parameters compared to the previous example.

```bash
torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.train_dc_ae_diffusion_model resolution=512 \
    train_dataset=latent_imagenet latent_imagenet.batch_size=128 latent_imagenet.data_dir=assets/data/latent/dc_ae_f32c32_in_1.0/imagenet_512 \
    evaluate_dataset=sample_class sample_class.num_samples=50000 \
    autoencoder=dc-ae-f32c32-in-1.0 scaling_factor=0.3189 \
    model=dit dit.learn_sigma=True dit.in_channels=32 dit.patch_size=1 dit.depth=28 dit.hidden_size=1152 dit.num_heads=16 \
    dit.train_scheduler=GaussianDiffusion dit.eval_scheduler=GaussianDiffusion \
    optimizer.name=adamw optimizer.lr=0.0002 optimizer.weight_decay=0 optimizer.betas=[0.9,0.999] lr_scheduler.name=constant amp=fp16 \
    max_steps=3000000 ema_decay=0.9999 \
    fid.ref_path=assets/data/fid/imagenet_512_train.npz \
    run_dir=.exp/diffusion/imagenet_512/dc_ae_f32c32_in_1.0/dit_xl_1/bs_1024_lr_2e-4_fp16 log=False
```

--------------------------------

### Run EfficientViT-SAM Interactive Demo

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Launches an interactive demo for EfficientViT-SAM. Supports segmenting everything, point-prompted segmentation, and box-prompted segmentation with specified coordinates.

```bash
python applications/efficientvit_sam/demo_efficientvit_sam_model.py \
    --model efficientvit-sam-xl1 --mode all
```

```bash
python applications/efficientvit_sam/demo_efficientvit_sam_model.py \
    --model efficientvit-sam-xl1 --mode point
```

```bash
python applications/efficientvit_sam/demo_efficientvit_sam_model.py \
    --model efficientvit-sam-xl1 --mode box --box "[150,70,640,400]"
```

--------------------------------

### Visualize EfficientViT-SAM Model Outputs

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Launches a demo script to visualize the outputs of the EfficientViT-SAM models. Supports different visualization modes like 'all', 'point', and 'box'.

```bash
python applications/efficientvit_sam/demo_efficientvit_sam_model.py --model efficientvit-sam-xl1 --mode all
```

```bash
python applications/efficientvit_sam/demo_efficientvit_sam_model.py --model efficientvit-sam-xl1 --mode point
```

```bash
python applications/efficientvit_sam/demo_efficientvit_sam_model.py --model efficientvit-sam-xl1 --mode box --box "[150,70,640,400]"
```

--------------------------------

### Run DC-AE-Diffusion Demo via CLI

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Launches a command-line demo for the DC-AE-Diffusion model. Specify the model name and a run directory for output.

```bash
# Single-GPU demo
torchrun --nnodes=1 --nproc_per_node=1 \
    -m applications.dc_ae.demo_dc_ae_diffusion_model \
    model=dc-ae-f64c128-in-1.0-uvit-h-in-512px-train2000k \
    run_dir=.demo/diffusion/dc-ae-f64c128-in-1.0-uvit-h-in-512px-train2000k
```

--------------------------------

### Demo DC-AE-Diffusion Model

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Run a demo for a DC-AE-Diffusion model. This command launches a demonstration of the model's capabilities.

```bash
torchrun --nnodes=1 --nproc_per_node=1 -m applications.dc_ae.demo_dc_ae_diffusion_model model=dc-ae-f64c128-in-1.0-uvit-h-in-512px-train2000k run_dir=.demo/diffusion/dc-ae-f64c128-in-1.0-uvit-h-in-512px-train2000k
```

--------------------------------

### Visualize EfficientViT Segmentation Models

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_seg/README.md

Run this script to visualize segmentation models. Specify the image path, dataset, crop size, and model name.

```bash
python applications/efficientvit_seg/demo_efficientvit_seg_model.py --image_path assets/fig/indoor.jpg --dataset ade20k --crop_size 512 --model efficientvit-seg-l2-ade20k
```

```bash
python applications/efficientvit_seg/demo_efficientvit_seg_model.py --image_path assets/fig/city.png --dataset cityscapes --crop_size 1024 --model efficientvit-seg-l2-cityscapes
```

--------------------------------

### Train EfficientViT-SAM Model

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Use this command to initiate the training process for EfficientViT-SAM models. Ensure distilled models are downloaded and paths are correctly set.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/train_efficientvit_sam_model.py applications/efficientvit_sam/configs/efficientvit_sam_xl1.yaml --data_provider.root ~/dataset/sam/ --path .exp/efficientvit_sam/efficientvit_sam_xl1 --resume
```

--------------------------------

### Run EfficientViT-GazeSAM Demo with TensorRT

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Launches the EfficientViT-GazeSAM demo in TensorRT mode for real-time performance, optimized for GPUs like RTX 4070. This command enables webcam input by default.

```bash
# TensorRT mode — webcam (default, real-time on RTX 4070)
python applications/efficientvit_gazesam/gazesam_demo.py --webcam
```

--------------------------------

### Run GazeSAM with Video File

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_gazesam/README.md

Launches the GazeSAM demo using a specified video file as input. By default, it uses TensorRT for real-time performance. Results are saved to the `output_videos` directory.

```bash
python gazesam_demo.py --video <path>
```

--------------------------------

### Run GazeSAM with Webcam

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_gazesam/README.md

Launches the GazeSAM demo using the webcam as input. By default, it uses TensorRT for real-time performance. Results are saved to the `output_videos` directory.

```bash
python gazesam_demo.py --webcam
```

--------------------------------

### DC-AE-Diffusion Demo (CLI)

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Generate images from a pretrained DC-AE+UViT model via the command-line demo script.

```APIDOC
## DC-AE-Diffusion Demo (CLI)

### Description
Generate images from a pretrained DC-AE+UViT model via the command-line demo script.

### Command
```bash
torchrun --nnodes=1 --nproc_per_node=1 \
    -m applications.dc_ae.demo_dc_ae_diffusion_model \
    model=dc-ae-f64c128-in-1.0-uvit-h-in-512px-train2000k \
    run_dir=.demo/diffusion/dc-ae-f64c128-in-1.0-uvit-h-in-512px-train2000k
```
```

--------------------------------

### Run GazeSAM Demo with Default TensorRT Engines

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Execute the GazeSAM demo using a video file with default TensorRT engines. Ensure the input video path is correctly specified.

```bash
python applications/efficientvit_gazesam/gazesam_demo.py \
    --video input_videos/example.mp4 \
    --precision-mode default
```

--------------------------------

### Run GazeSAM with Webcam and Optimized Engines

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_gazesam/README.md

Executes the GazeSAM demo using the webcam and the optimized TensorRT precision mode (FP32, FP16, and INT8 engines) for potentially faster performance.

```bash
python gazesam_demo.py --webcam --precision-mode optimized
```

--------------------------------

### Build and Use DC-AE-Diffusion Models

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Load a pre-trained DC-AE Diffusion model from Hugging Face, generate latent samples using prompts, and decode them into images. Ensure CUDA is available and models are moved to the appropriate device.

```python
# build DC-AE-Diffusion models
# full DC-AE-Diffusion model list: https://huggingface.co/collections/mit-han-lab/dc-ae-diffusion-670dbb8d6b6914cf24c1a49d
from efficientvit.diffusion_model_zoo import DCAE_Diffusion_HF

dc_ae_diffusion = DCAE_Diffusion_HF.from_pretrained(f"mit-han-lab/dc-ae-f64c128-in-1.0-uvit-h-in-512px-train2000k")

# denoising on the latent space
import torch
import numpy as np
from torchvision.utils import save_image

torch.set_grad_enabled(False)
device = torch.device("cuda")
dc_ae_diffusion = dc_ae_diffusion.to(device).eval()

seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
eval_generator = torch.Generator(device=device)
eval_generator.manual_seed(seed)

prompts = torch.tensor(
    [279, 333, 979, 936, 933, 145, 497, 1, 248, 360, 793, 12, 387, 437, 938, 978], dtype=torch.int, device=device
)
num_samples = prompts.shape[0]
prompts_null = 1000 * torch.ones((num_samples,), dtype=torch.int, device=device)
latent_samples = dc_ae_diffusion.diffusion_model.generate(prompts, prompts_null, 6.0, eval_generator)
latent_samples = latent_samples / dc_ae_diffusion.scaling_factor

# decode
image_samples = dc_ae_diffusion.autoencoder.decode(latent_samples)
save_image(image_samples * 0.5 + 0.5, "demo_dc_ae_diffusion.png", nrow=int(np.sqrt(num_samples)))

```

--------------------------------

### Run GazeSAM Demo with ONNX Runtime

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Execute the GazeSAM demo using a video file with ONNX runtime as a fallback. This is useful when TensorRT is not available or preferred.

```bash
python applications/efficientvit_gazesam/gazesam_demo.py \
    --video input_videos/example.mp4 \
    --runtime onnx
```

--------------------------------

### Run GazeSAM with Input Video and Default Engines

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_gazesam/README.md

Executes the GazeSAM demo with a specific input video file and uses the default TensorRT precision mode (FP32 + FP16 engines).

```bash
python gazesam_demo.py --video input_videos/example.mp4 --precision-mode default
```

--------------------------------

### Load Pretrained EfficientViT-SAM Model

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Instantiates a pretrained EfficientViT-SAM model using a factory function. Supports multiple variants and resolutions. Ensure CUDA is available and the model is moved to the GPU.

```python
import numpy as np
from PIL import Image
import torch
from efficientvit.sam_model_zoo import create_efficientvit_sam_model
from efficientvit.models.efficientvit.sam import (
    EfficientViTSamPredictor,
    EfficientViTSamAutomaticMaskGenerator,
)

# Available: efficientvit-sam-l0, l1, l2, xl0, xl1
efficientvit_sam = create_efficientvit_sam_model(
    name="efficientvit-sam-xl1", pretrained=True
).cuda().eval()
```

--------------------------------

### Evaluate EfficientViT-SAM on COCO with Point Prompts

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Evaluates the EfficientViT-SAM model on the COCO dataset using single point prompts. Specify the number of clicks for prompting.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py --dataset coco --image_root ~/dataset/coco/val2017 --annotation_json_file ~/dataset/coco/annotations/instances_val2017.json --model efficientvit-sam-xl1 --prompt_type point --num_click 1
```

--------------------------------

### Download SD-VAE

Source: https://github.com/mit-han-lab/efficientvit/blob/master/assets/train_diffusion_with_sd_vae_f8.md

Use this command to download the pre-trained Stable Diffusion VAE (sd-vae-ft-ema) using the diffusers library.

```bash
python -c "import diffusers; diffusers.models.AutoencoderKL.from_pretrained('stabilityai/sd-vae-ft-ema')"
```

--------------------------------

### Point-Prompted Segmentation with EfficientViT-SAM

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Performs point-prompted instance segmentation using EfficientViT-SAM. Requires setting the image, providing point coordinates and labels, and optionally specifying multimask output.

```python
# --- Point-prompted segmentation ---
predictor = EfficientViTSamPredictor(efficientvit_sam)
image = np.array(Image.open("assets/fig/cat.jpg").convert("RGB"))
predictor.set_image(image)

masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),   # (x, y) pixel coordinates
    point_labels=np.array([1]),            # 1 = foreground
    multimask_output=True,
)
# masks: (3, H, W) bool array; scores: (3,) confidence per mask
best = masks[scores.argmax()]
print("Best mask shape:", best.shape, " Score:", scores.max())
```

--------------------------------

### Evaluate EfficientViT-SAM on LVIS with Point Prompts

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Evaluates the EfficientViT-SAM model on the LVIS dataset using single point prompts. Specify the number of clicks for prompting.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py --dataset lvis --image_root ~/dataset/coco --annotation_json_file ~/dataset/coco/annotations/lvis_v1_val.json --model efficientvit-sam-xl1 --prompt_type point --num_click 1
```

--------------------------------

### Evaluate EfficientViT-SAM with Point Prompts on LVIS

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Evaluates the EfficientViT-SAM model on the LVIS dataset using point prompts with a single click. Ensure the dataset path and annotation file are correct.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py \
    --dataset lvis \
    --image_root ~/dataset/coco \
    --annotation_json_file ~/dataset/coco/annotations/lvis_v1_val.json \
    --model efficientvit-sam-xl1 \
    --prompt_type point --num_click 1
```

--------------------------------

### Load EfficientViT Classification Model

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Demonstrates how to import and use the `create_efficientvit_cls_model` function to load EfficientViT image classification models. Available models include B-series for edge and L-series for server.

```python
import torch
from PIL import Image
import torchvision.transforms as T
from efficientvit.cls_model_zoo import create_efficientvit_cls_model

# Available models (examples):
#   B-series: efficientvit-b1-r224, b2-r256, b3-r288
```

--------------------------------

### Run GazeSAM Demo with Optimized TensorRT Engines

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Run the GazeSAM demo using webcam input with optimized TensorRT engines for faster inference. This mode is approximately 5ms faster per frame.

```bash
python applications/efficientvit_gazesam/gazesam_demo.py \
    --webcam \
    --precision-mode optimized
```

--------------------------------

### Run DC-AE Model Reconstruction from Command Line

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Execute the DC-AE model for image reconstruction directly from the command line, specifying the model name, run directory, and input image path list.

```bash
python -m applications.dc_ae.demo_dc_ae_model_diffusers model=dc-ae-f32c32-in-1.0-diffusers run_dir=.demo/reconstruction/dc-ae-f32c32-in-1.0-diffusers input_path_list=[assets/fig/girl.png]
```

--------------------------------

### EfficientViT Segmentation Demo and Export Commands

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Commands for visualizing segmentation predictions and exporting models to ONNX/TFLite. Ensure the dataset, model, and resolution parameters are set correctly.

```bash
# Visualization (ADE20K)
python applications/efficientvit_seg/demo_efficientvit_seg_model.py \
    --image_path assets/fig/indoor.jpg \
    --dataset ade20k \
    --crop_size 512 \
    --model efficientvit-seg-l2-ade20k

# Visualization (Cityscapes)
python applications/efficientvit_seg/demo_efficientvit_seg_model.py \
    --image_path assets/fig/city.png \
    --dataset cityscapes \
    --crop_size 1024 \
    --model efficientvit-seg-l2-cityscapes

# ONNX export (Cityscapes 1024×2048)
python assets/onnx_export.py \
    --export_path assets/export_models/efficientvit_seg_l2_cityscapes_r1024x2048.onnx \
    --task seg \
    --model efficientvit-seg-l2-cityscapes \
    --resolution 1024 2048 \
    --bs 1

# TFLite export (ADE20K 512×512)
python assets/tflite_export.py \
    --export_path assets/export_models/efficientvit_seg_l2_ade20k_r512x512.tflite \
    --task seg \
    --model efficientvit-seg-l2-ade20k \
    --resolution 512 512
```

--------------------------------

### Evaluate EfficientViT-SAM on LVIS with Box Prompts

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Evaluates the EfficientViT-SAM model on the LVIS dataset using bounding box prompts. Ensure the model and dataset paths are correctly configured.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py --dataset lvis --image_root ~/dataset/coco --annotation_json_file ~/dataset/coco/annotations/lvis_v1_val.json --model efficientvit-sam-xl1 --prompt_type box
```

--------------------------------

### Create EfficientViT Segmentation Model

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_seg/README.md

Instantiates a pretrained EfficientViT segmentation model for either Cityscapes or ADE20K datasets. Ensure the model name and pretrained flag are set correctly.

```python
from efficientvit.seg_model_zoo import create_efficientvit_seg_model

model = create_efficientvit_seg_model(name="efficientvit-seg-l2-cityscapes", pretrained=True)

model = create_efficientvit_seg_model(name="efficientvit-seg-l2-ade20k", pretrained=True)
```

--------------------------------

### Evaluate EfficientViT-SAM on COCO with Box Prompts from Detector

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Evaluates the EfficientViT-SAM model on the COCO dataset using bounding box prompts generated by a detector. Requires a source JSON file containing detected boxes.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py --dataset coco --image_root ~/dataset/coco/val2017 --annotation_json_file ~/dataset/coco/annotations/instances_val2017.json --model efficientvit-sam-xl1 --prompt_type box_from_detector --source_json_file ~/dataset/coco/source_json_file/coco_vitdet.json
```

--------------------------------

### Train EfficientViT Classification Models

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Commands for training EfficientViT classification models on ImageNet using distributed data parallel. Supports bf16 AMP and multi-resolution training.

```bash
# L-series (bf16 AMP)
torchrun --nnodes 1 --nproc_per_node=8 \
    python applications/efficientvit_cls/train_efficientvit_cls_model.py \
    applications/efficientvit_cls/configs/imagenet/efficientvit_l3.yaml \
    --amp bf16 \
    --data_provider.data_dir ~/dataset/imagenet \
    --path .exp/efficientvit_cls/imagenet/efficientvit_l3_r224/

# B-series with multi-resolution training (random resolution augmentation)
torchrun --nnodes 1 --nproc_per_node=8 \
    python applications/efficientvit_cls/train_efficientvit_cls_model.py \
    applications/efficientvit_cls/configs/imagenet/efficientvit_b2.yaml \
    --data_provider.image_size "[128,160,192,224,256,288]" \
    --data_provider.data_dir ~/dataset/imagenet \
    --run_config.eval_image_size "[288]" \
    --data_provider.data_aug "{n:1,m:5}" \
    --path .exp/efficientvit_cls/imagenet/efficientvit_b2_r288/
```

--------------------------------

### create_efficientvit_sam_model

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Factory function that instantiates a pretrained EfficientViT-SAM model. Supports five variants (L0, L1, L2, XL0, XL1) at resolutions 512×512 or 1024×1024.

```APIDOC
## `create_efficientvit_sam_model` — Load EfficientViT-SAM

### Description
Factory function that instantiates a pretrained EfficientViT-SAM model. Supports five variants (L0, L1, L2, XL0, XL1) at resolutions 512×512 or 1024×1024.

### Method Signature
```python
create_efficientvit_sam_model(name: str, pretrained: bool)
```

### Parameters
- **name** (str) - The name of the EfficientViT-SAM variant (e.g., "efficientvit-sam-xl1").
- **pretrained** (bool) - Whether to load pretrained weights.

### Request Example
```python
import torch
from efficientvit.sam_model_zoo import create_efficientvit_sam_model

# Available: efficientvit-sam-l0, l1, l2, xl0, xl1
efficientvit_sam = create_efficientvit_sam_model(
    name="efficientvit-sam-xl1", pretrained=True
).cuda().eval()
```
```

--------------------------------

### Evaluate DC-AE-Diffusion Model with CFG

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Evaluate DC-AE-Diffusion models with classifier-free guidance (CFG). Adjust cfg_scale based on the model for optimal results.

```bash
# full DC-AE-Diffusion model list: https://huggingface.co/collections/mit-han-lab/dc-ae-diffusion-670dbb8d6b6914cf24c1a49d
# cfg=1.3 for mit-han-lab/dc-ae-f32c32-in-1.0-dit-xl-in-512px
# and cfg=1.5 for all other models
torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.eval_dc_ae_diffusion_model dataset=imagenet_512 model=dc-ae-f64c128-in-1.0-uvit-h-in-512px cfg_scale=1.5 run_dir=tmp
# Expected results:
#   fid: 2.963459255529642

torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.eval_dc_ae_diffusion_model dataset=imagenet_512 model=dc-ae-f32c32-in-1.0-usit-h-in-512px cfg_scale=1.2 run_dir=tmp
# Expected results:
#   fid: 1.883184155951767

torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.eval_dc_ae_diffusion_model dataset=imagenet_512 model=dc-ae-f32c32-in-1.0-usit-2b-in-512px cfg_scale=1.15 run_dir=tmp
# Expected results:
#   fid: 1.7210562991380698
```

--------------------------------

### Evaluate EfficientViT-SAM on LVIS with Box Prompts from Detector

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Evaluates the EfficientViT-SAM model on the LVIS dataset using bounding box prompts generated by a detector. Requires a source JSON file containing detected boxes.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py --dataset lvis --image_root ~/dataset/coco --annotation_json_file ~/dataset/coco/annotations/lvis_v1_val.json --model efficientvit-sam-xl1 --prompt_type box_from_detector --source_json_file ~/dataset/coco/source_json_file/lvis_vitdet.json
```

--------------------------------

### Evaluate EfficientViT-SAM with Box Prompts on COCO

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Evaluates the EfficientViT-SAM model on the COCO dataset using ground-truth boxes for prompting. Ensure the dataset and model are correctly configured.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py \
    --dataset coco \
    --image_root ~/dataset/coco/val2017 \
    --annotation_json_file ~/dataset/coco/annotations/instances_val2017.json \
    --model efficientvit-sam-xl1 \
    --prompt_type box
```

--------------------------------

### Instantiate EfficientViT Classification Model

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_cls/README.md

Use this function to create an EfficientViT classification model. Specify the model name and whether to load pretrained weights.

```python
from efficientvit.cls_model_zoo import create_efficientvit_cls_model

model = create_efficientvit_cls_model(name="efficientvit-l3-r384", pretrained=True)
```

--------------------------------

### Evaluate Box-Prompted Instance Segmentation on COCO

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Runs evaluation for box-prompted zero-shot instance segmentation on the COCO dataset using a specified EfficientViT-SAM model. Requires dataset paths and annotation files. The prompt type is set to 'box'.

```bash
# COCO
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py --dataset coco --image_root ~/dataset/coco/val2017 --annotation_json_file ~/dataset/coco/annotations/instances_val2017.json --model efficientvit-sam-xl1 --prompt_type box

# expected results: all=79.927, large=83.748, medium=82.210, small=75.833
```

--------------------------------

### Box-Prompted Segmentation with EfficientViT-SAM

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Performs box-prompted instance segmentation using EfficientViT-SAM. Provide the bounding box coordinates and set multimask_output to False for a single mask.

```python
# --- Box-prompted segmentation ---
masks_box, _, _ = predictor.predict(
    box=np.array([150, 70, 640, 400]),     # [x1, y1, x2, y2]
    multimask_output=False,
)
```

--------------------------------

### Train EfficientViT B2 Model (Variable Resolution and Augmentation)

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_cls/README.md

Train the EfficientViT B2 model on ImageNet with variable image sizes, specific data augmentation, and a defined evaluation image size. Uses PyTorch distributed training. Ensure the dataset is available at the specified data directory.

```bash
torchrun --nnodes 1 --nproc_per_node=8 \
python applications/efficientvit_cls/train_efficientvit_cls_model.py applications/efficientvit_cls/configs/imagenet/efficientvit_b2.yaml \
    --data_provider.image_size "[128,160,192,224,256,288]" \
    --data_provider.data_dir ~/dataset/imagenet \
    --run_config.eval_image_size "[288]" \
    --data_provider.data_aug "{n:1,m:5}" \
    --path .exp/efficientvit_cls/imagenet/efficientvit_b2_r288/
```

--------------------------------

### Load and Run EfficientViT Segmentation Model

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Loads a pre-trained EfficientViT segmentation model and performs inference on an image. Ensure the model name and dataset (Cityscapes/ADE20K) are correct.

```python
import torch
import numpy as np
from PIL import Image
import torchvision.transforms as T
from efficientvit.seg_model_zoo import create_efficientvit_seg_model

# Available models:
#   Cityscapes: efficientvit-seg-b0/b1/b2/b3/l1/l2-cityscapes  (1024×2048)
#   ADE20K:     efficientvit-seg-b1/b2/b3/l1/l2-ade20k          (512×512)
model = create_efficientvit_seg_model(
    name="efficientvit-seg-l2-ade20k", pretrained=True
).cuda().eval()

transform = T.Compose([
    T.Resize((512, 512)),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

image = Image.open("assets/fig/indoor.jpg").convert("RGB")
x = transform(image)[None].cuda()       # (1, 3, 512, 512)

with torch.no_grad():
    output = model(x)                   # (1, num_classes, 512, 512)
    pred = output.argmax(dim=1)         # (1, 512, 512)
    print("Predicted label map shape:", pred.shape)
# EfficientViT-L2 ADE20K: 50.702 mIoU
```

--------------------------------

### Generate Reference for FID Computation

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Generate a reference dataset for FID computation. This is a prerequisite for evaluating diffusion models.

```bash
# generate reference for FID computation
torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.generate_reference \
    dataset=imagenet imagenet.resolution=512 imagenet.image_mean=[0.,0.,0.] imagenet.image_std=[1.,1.,1.] split=train \
    fid.save_path=assets/data/fid/imagenet_512_train.npz
```

--------------------------------

### Evaluate EfficientViT-SAM with Detector Predicted Boxes on COCO

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Evaluates the EfficientViT-SAM model on the COCO dataset using boxes predicted by a detector (ViTDet). Requires a source JSON file containing detector predictions.

```bash
torchrun --nproc_per_node=8 applications/efficientvit_sam/eval_efficientvit_sam_model.py \
    --dataset coco \
    --image_root ~/dataset/coco/val2017 \
    --annotation_json_file ~/dataset/coco/annotations/instances_val2017.json \
    --model efficientvit-sam-xl1 \
    --prompt_type box_from_detector \
    --source_json_file ~/dataset/coco/source_json_file/coco_vitdet.json
```

--------------------------------

### Run ONNX Inference with EfficientViT-SAM

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Performs inference using EfficientViT-SAM models exported to ONNX format. Requires paths to the encoder and decoder ONNX files and the desired mode (e.g., 'point').

```bash
python applications/efficientvit_sam/run_efficientvit_sam_onnx.py \
    --model efficientvit-sam-xl1 \
    --encoder_model assets/export_models/efficientvit_sam/onnx/efficientvit_sam_xl1_encoder.onnx \
    --decoder_model assets/export_models/efficientvit_sam/onnx/efficientvit_sam_xl1_decoder.onnx \
    --mode point
```

--------------------------------

### Train DC-AE-Diffusion Model

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Step 2 of training: Trains a DC-AE + USiT or UViT diffusion model using pre-generated latents. Configures dataset, model architecture, optimizer, and learning rate scheduler.

```bash
# Step 2: Train DC-AE-f32 + USiT-H on ImageNet 512x512
torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.train_dc_ae_diffusion_model \
    resolution=512 \
    train_dataset=latent_imagenet \
    latent_imagenet.batch_size=128 \
    latent_imagenet.data_dir=assets/data/latent/dc_ae_f32c32_in_1.0/imagenet_512 \
    evaluate_dataset=sample_class \
    sample_class.num_samples=50000 \
    autoencoder=dc-ae-f32c32-in-1.0 scaling_factor=0.3189 \
    model=uvit uvit.depth=28 uvit.hidden_size=1152 uvit.num_heads=16 \
    uvit.in_channels=32 uvit.patch_size=1 \
    uvit.train_scheduler=SiTSampler uvit.eval_scheduler=ODE_dopri5 \
    uvit.num_inference_steps=250 \
    optimizer.name=adamw optimizer.lr=1e-4 optimizer.weight_decay=0 \
    optimizer.betas=[0.99,0.99] \
    lr_scheduler.name=constant_with_warmup lr_scheduler.warmup_steps=5000 \
    amp=bf16 max_steps=500000 ema_decay=0.9999 \
    fid.ref_path=assets/data/fid/imagenet_512_train.npz \
    run_dir=.exp/diffusion/imagenet_512/dc_ae_f32c32_in_1.0/usit_h/bs_1024_lr_1e-4_bf16 \
    log=False
```

--------------------------------

### Load and Run EfficientViT Classification Model

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Loads a pre-trained EfficientViT classification model and performs inference on an image. Ensure the model name and resolution match your requirements.

```python
model = create_efficientvit_cls_model(
    name="efficientvit-l3-r384", pretrained=True
).cuda().eval()

transform = T.Compose([
    T.Resize(384),
    T.CenterCrop(384),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

image = Image.open("assets/fig/cat.jpg").convert("RGB")
x = transform(image)[None].cuda()          # (1, 3, 384, 384)

with torch.no_grad():
    logits = model(x)                      # (1, 1000)
    probs = logits.softmax(dim=-1)
    top5 = probs.topk(5)
    print("Top-5 class indices:", top5.indices[0].tolist())
    print("Top-5 probabilities:", top5.values[0].tolist())
# EfficientViT-L3 @ 384: 86.408 Top-1 Acc on ImageNet-1K
```

--------------------------------

### Load EfficientViT-SAM Model

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Loads a pretrained EfficientViT-SAM model. Ensure the model name is valid and pretrained weights are available. The model is moved to CUDA and set to evaluation mode.

```python
from efficientvit.sam_model_zoo import create_efficientvit_sam_model

efficientvit_sam = create_efficientvit_sam_model(name="efficientvit-sam-xl1", pretrained=True)
efficientvit_sam = efficientvit_sam.cuda().eval()
```

--------------------------------

### Initialize EfficientViTSamPredictor

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Initializes the EfficientViTSamPredictor with a loaded EfficientViT-SAM model. This predictor is used for single-instance segmentation based on prompts.

```python
from efficientvit.models.efficientvit.sam import EfficientViTSamPredictor

efficientvit_sam_predictor = EfficientViTSamPredictor(efficientvit_sam)
```

--------------------------------

### Train DC-AE-f64 + UViT-H on ImageNet 512x512

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Use this command to train a diffusion model with DC-AE-f64 and UViT-H on ImageNet at 512x512 resolution. It specifies dataset paths, autoencoder configuration, model parameters for UViT, and training hyperparameters.

```bash
torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.train_dc_ae_diffusion_model resolution=512 \
    train_dataset=latent_imagenet latent_imagenet.batch_size=128 latent_imagenet.data_dir=assets/data/latent/dc_ae_f64c128_in_1.0/imagenet_512 \
    evaluate_dataset=sample_class sample_class.num_samples=50000 \
    autoencoder=dc-ae-f64c128-in-1.0 scaling_factor=0.2889 \
    model=uvit uvit.depth=28 uvit.hidden_size=1152 uvit.num_heads=16 uvit.in_channels=128 uvit.patch_size=1 \
    uvit.train_scheduler=DPM_Solver uvit.eval_scheduler=DPM_Solver \
    optimizer.name=adamw optimizer.lr=2e-4 optimizer.weight_decay=0.03 optimizer.betas=[0.99,0.99] lr_scheduler.name=constant_with_warmup lr_scheduler.warmup_steps=5000 amp=bf16 \
    max_steps=500000 ema_decay=0.9999 \
    fid.ref_path=assets/data/fid/imagenet_512_train.npz \
    run_dir=.exp/diffusion/imagenet_512/dc_ae_f64c128_in_1.0/uvit_h_1/bs_1024_lr_2e-4_bf16 log=False
```

--------------------------------

### Evaluate DC-AE-Diffusion Model without CFG

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Evaluate DC-AE-Diffusion models without classifier-free guidance (CFG). This helps assess baseline performance.

```bash
# full DC-AE-Diffusion model list: https://huggingface.co/collections/mit-han-lab/dc-ae-diffusion-670dbb8d6b6914cf24c1a49d

torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.eval_dc_ae_diffusion_model dataset=imagenet_512 model=dc-ae-f64c128-in-1.0-uvit-h-in-512px cfg_scale=1.0 run_dir=tmp
# Expected results:
#   fid: 13.754458694549271

torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.eval_dc_ae_diffusion_model dataset=imagenet_512 model=dc-ae-f32c32-in-1.0-usit-h-in-512px cfg_scale=1.0 run_dir=tmp
# Expected results:
#   fid: 3.791360749566877
```

--------------------------------

### Perform ONNX Inference with EfficientViT-SAM

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_sam/README.md

Runs inference using the ONNX exported EfficientViT-SAM encoder and decoder models. Requires paths to the ONNX engine files and specifies the inference mode.

```bash
python applications/efficientvit_sam/run_efficientvit_sam_onnx.py --model efficientvit-sam-xl1 --encoder_model assets/export_models/efficientvit_sam/onnx/efficientvit_sam_xl1_encoder.onnx --decoder_model assets/export_models/efficientvit_sam/onnx/efficientvit_sam_xl1_decoder.onnx --mode point
```

--------------------------------

### Generate Images with DC-AE-Diffusion

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Loads a pretrained DC-AE-Diffusion model and generates image samples based on class prompts. Ensure the device is set and necessary libraries like torch and numpy are imported.

```python
dc_ae_diffusion = DCAE_Diffusion_HF.from_pretrained(
    "mit-han-lab/dc-ae-f64c128-in-1.0-uvit-h-in-512px-train2000k"
).to(device).eval()

seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
eval_generator = torch.Generator(device=device)
eval_generator.manual_seed(seed)

# 16 ImageNet class labels (e.g., 279=fox, 333=hamster, 979=valley…)
prompts = torch.tensor(
    [279, 333, 979, 936, 933, 145, 497, 1, 248, 360, 793, 12, 387, 437, 938, 978],
    dtype=torch.int, device=device,
)
num_samples = prompts.shape[0]
prompts_null = 1000 * torch.ones((num_samples,), dtype=torch.int, device=device)

# Classifier-free guidance scale = 6.0
latent_samples = dc_ae_diffusion.diffusion_model.generate(
    prompts, prompts_null, 6.0, eval_generator
)
latent_samples = latent_samples / dc_ae_diffusion.scaling_factor

image_samples = dc_ae_diffusion.autoencoder.decode(latent_samples)
save_image(image_samples * 0.5 + 0.5, "demo_diffusion.png", nrow=int(np.sqrt(num_samples)))
```

--------------------------------

### Train EfficientViT B1 Model (Variable Resolution)

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_cls/README.md

Train the EfficientViT B1 model on ImageNet with variable image sizes and a specific evaluation image size. Uses PyTorch distributed training. Ensure the dataset is available at the specified data directory.

```bash
torchrun --nnodes 1 --nproc_per_node=8 \
python applications/efficientvit_cls/train_efficientvit_cls_model.py applications/efficientvit_cls/configs/imagenet/efficientvit_b1.yaml \
    --data_provider.image_size "[128,160,192,224,256,288]" \
    --data_provider.data_dir ~/dataset/imagenet \
    --run_config.eval_image_size "[288]" \
    --path .exp/efficientvit_cls/imagenet/efficientvit_b1_r288/
```

--------------------------------

### Train EfficientViT B3 Model (224x224)

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_cls/README.md

Train the EfficientViT B3 model on ImageNet with a resolution of 224x224. Uses PyTorch distributed training. Ensure the dataset is available at the specified data directory.

```bash
torchrun --nnodes 1 --nproc_per_node=8 \
python applications/efficientvit_cls/train_efficientvit_cls_model.py applications/efficientvit_cls/configs/imagenet/efficientvit_b3.yaml \
    --data_provider.data_dir ~/dataset/imagenet \
    --path .exp/efficientvit_cls/imagenet/efficientvit_b3_r224/
```

--------------------------------

### DC-AE-Diffusion Training - Step 1: Pre-generate latents

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Pre-generate and cache latents for DC-AE diffusion model training.

```APIDOC
## DC-AE-Diffusion Training - Step 1: Pre-generate latents

### Description
Pre-generate and cache latents for DC-AE diffusion model training.

### Command
```bash
torchrun --nnodes=1 --nproc_per_node=8 -m applications.dc_ae.dc_ae_generate_latent \
    resolution=512 \
    image_root_path=~/dataset/imagenet/train \
    batch_size=64 \
    model_name=dc-ae-f32c32-in-1.0 \
    scaling_factor=0.3189 \
    latent_root_path=assets/data/latent/dc_ae_f32c32_in_1.0/imagenet_512
```
```

--------------------------------

### Reconstruct Image using DC-AE via Command Line

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/dc_ae/README.md

Reconstructs an image using a specified DC-AE model via the command line. This script requires the model name, a run directory, and a list of input image paths.

```bash
python -m applications.dc_ae.demo_dc_ae_model model=dc-ae-f32c32-in-1.0 run_dir=.demo/reconstruction/dc-ae-f32c32-in-1.0 input_path_list=[assets/fig/girl.png]
```

--------------------------------

### Train EfficientViT B1 Model (224x224)

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_cls/README.md

Train the EfficientViT B1 model on ImageNet with a resolution of 224x224. Uses PyTorch distributed training. Ensure the dataset is available at the specified data directory.

```bash
torchrun --nnodes 1 --nproc_per_node=8 \
python applications/efficientvit_cls/train_efficientvit_cls_model.py applications/efficientvit_cls/configs/imagenet/efficientvit_b1.yaml \
    --data_provider.data_dir ~/dataset/imagenet \
    --path .exp/efficientvit_cls/imagenet/efficientvit_b1_r224/
```

--------------------------------

### Train EfficientViT B2 Model (224x224)

Source: https://github.com/mit-han-lab/efficientvit/blob/master/applications/efficientvit_cls/README.md

Train the EfficientViT B2 model on ImageNet with a resolution of 224x224. Uses PyTorch distributed training. Ensure the dataset is available at the specified data directory.

```bash
torchrun --nnodes 1 --nproc_per_node=8 \
python applications/efficientvit_cls/train_efficientvit_cls_model.py applications/efficientvit_cls/configs/imagenet/efficientvit_b2.yaml \
    --data_provider.data_dir ~/dataset/imagenet \
    --path .exp/efficientvit_cls/imagenet/efficientvit_b2_r224/
```

--------------------------------

### Run TensorRT Inference with EfficientViT-SAM

Source: https://context7.com/mit-han-lab/efficientvit/llms.txt

Performs inference using EfficientViT-SAM models compiled to TensorRT engines. Requires paths to the encoder and decoder TensorRT engines and the desired mode.

```bash
python applications/efficientvit_sam/run_efficientvit_sam_trt.py \
    --model efficientvit-sam-xl1 \
    --encoder_engine assets/export_models/efficientvit_sam/tensorrt/efficientvit_sam_xl1_encoder.engine \
    --decoder_engine assets/export_models/efficientvit_sam/tensorrt/efficientvit_sam_xl1_decoder.engine \
    --mode point
```