TinyCLIP

https://github.com/wkcn/tinyclip
Admin
TinyCLIP is a cross-modal distillation method for large-scale language-image pre-trained models that...
Tokens:4,792
Snippets:40
Trust Score:8.3
License:MIT
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Install dependencies

Source: https://github.com/wkcn/tinyclip/blob/main/docs/PREPARATION.md

Install the training dependencies and the package in editable mode.

```bash
pip install -r requirements-training.txt
pip install -v -e .
```

--------------------------------

### Run manual weight inheritance training stages

Source: https://github.com/wkcn/tinyclip/blob/main/docs/PRETRAINING.md

Run the two-stage manual weight inheritance training for TinyCLIP, compressing OpenCLIP ViT-B/32 from 100% to 75% and then to 50% of the original size. Each script corresponds to one stage, and the next stage starts only after the previous one completes. Training uses 4 nodes, similar to the automatic weight inheritance setup.

```bash
sh script/manual_weight_inherit_100to75.sh # first stage
sh script/manual_weight_inherit_75to50.sh # second stage
```

--------------------------------

### Run auto weight inheritance training stages

Source: https://github.com/wkcn/tinyclip/blob/main/docs/PRETRAINING.md

Run the three-stage auto weight inheritance training for TinyCLIP, compressing OpenCLIP ViT-B/32 from 100% to 75%, then to 50%, and finally to 25% of the original size. Each script corresponds to one stage, and the next stage starts only after the previous one completes. Training uses 4 nodes with 8 GPUs per node.

```bash
sh script/auto_weight_inherit_100to75.sh # first stage
sh script/auto_weight_inherit_75to50.sh # second stage
sh script/auto_weight_inherit_50to25.sh # third stage
```

--------------------------------

### Evaluate TinyCLIP ViT-40M/32 + Text-19M (LAION-400M)

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the TinyCLIP ViT-40M/32 + Text-19M model trained on LAION-400M. Requires setting the paths to imagenet-val and resume.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model TinyCLIP-ViT-40M-32-Text-19M \
--eval \
--resume ./checkpoints/TinyCLIP-ViT-40M-32-Text-19M-LAION400M.pt
```

--------------------------------

### Evaluate TinyCLIP ResNet-19M + Text-19M (LAION-400M)

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the TinyCLIP ResNet-19M + Text-19M model trained on LAION-400M. Requires setting the paths to imagenet-val and resume.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model TinyCLIP-ResNet-19M-Text-19M \
--eval \
--resume ./checkpoints/TinyCLIP-ResNet-19M-Text-19M-LAION400M.pt
```

--------------------------------

### Evaluate TinyCLIP ViT-61M/32 + Text-29M (LAION-400M)

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the TinyCLIP ViT-61M/32 + Text-29M model trained on LAION-400M. Requires setting the paths to imagenet-val and resume.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model TinyCLIP-ViT-61M-32-Text-29M \
--eval \
--resume ./checkpoints/TinyCLIP-ViT-61M-32-Text-29M-LAION400M.pt
```

--------------------------------

### Evaluate TinyCLIP ResNet-30M + Text-29M (LAION-400M)

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the TinyCLIP ResNet-30M + Text-29M model trained on LAION-400M. Requires setting the paths to imagenet-val and resume.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model TinyCLIP-ResNet-30M-Text-29M \
--eval \
--resume ./checkpoints/TinyCLIP-ResNet-30M-Text-29M-LAION400M.pt
```

--------------------------------

### Evaluate TinyCLIP ViT-22M/32 + Text-10M (LAION-400M) auto

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the auto weight inference TinyCLIP ViT-22M/32 + Text-10M model trained on LAION-400M. Uses --prune-image and --prune-text flags.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model ViT-B-32 \
--prune-image \
--prune-text \
--eval \
--resume ./checkpoints/TinyCLIP-auto-ViT-22M-32-Text-10M-LAION400M.pt
```

--------------------------------

### Evaluate TinyCLIP ViT-45M/32 + Text-18M (LAION-400M) auto

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the auto weight inference TinyCLIP ViT-45M/32 + Text-18M model trained on LAION-400M. Uses --prune-image and --prune-text flags.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model ViT-B-32 \
--prune-image \
--prune-text \
--eval \
--resume ./checkpoints/TinyCLIP-auto-ViT-45M-32-Text-18M-LAION400M.pt
```

--------------------------------

### Evaluate TinyCLIP ViT-63M/32 + Text-31M (LAION-400M) auto

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the auto weight inference TinyCLIP ViT-63M/32 + Text-31M model trained on LAION-400M. Uses --prune-image and --prune-text flags.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model ViT-B-32 \
--prune-image \
--prune-text \
--eval \
--resume ./checkpoints/TinyCLIP-auto-ViT-63M-32-Text-31M-LAION400M.pt
```

--------------------------------

### Evaluate TinyCLIP ViT-45M/32 + Text-18M (LAION+YFCC-400M) auto

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the auto weight inference TinyCLIP ViT-45M/32 + Text-18M model trained on LAION+YFCC-400M. Uses --prune-image and --prune-text flags.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model ViT-B-32 \
--prune-image \
--prune-text \
--eval \
--resume ./checkpoints/TinyCLIP-auto-ViT-45M-32-Text-18M-LAIONYFCC400M.pt
```

--------------------------------

### Evaluate TinyCLIP ViT-63M/32 + Text-31M (LAION+YFCC-400M) auto

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the auto weight inference TinyCLIP ViT-63M/32 + Text-31M model trained on LAION+YFCC-400M. Uses --prune-image and --prune-text flags.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model ViT-B-32 \
--prune-image \
--prune-text \
--eval \
--resume ./checkpoints/TinyCLIP-auto-ViT-63M-32-Text-31M-LAIONYFCC400M.pt
```

--------------------------------

### Evaluate TinyCLIP ViT-8M/16 + Text-3M (YFCC-15M)

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the TinyCLIP ViT-8M/16 + Text-3M model trained on YFCC-15M. Requires setting the paths to imagenet-val and resume.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model TinyCLIP-ViT-8M-16-Text-3M \
--eval \
--resume ./checkpoints/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M.pt
```

--------------------------------

### Evaluate TinyCLIP ViT-39M/16 + Text-19M (YFCC-15M)

Source: https://github.com/wkcn/tinyclip/blob/main/docs/EVALUATION.md

Evaluate the TinyCLIP ViT-39M/16 + Text-19M model trained on YFCC-15M. Requires setting the paths to imagenet-val and resume.

```bash
python -m torch.distributed.launch --use_env --nproc_per_node 8 src/training/main_for_test.py \
--imagenet-val ./ImageNet \
--model TinyCLIP-ViT-39M-16-Text-19M \
--eval \
--resume ./checkpoints/TinyCLIP-ViT-39M-16-Text-19M-YFCC15M.pt
```

--------------------------------

### ImageNet-1k tar archive structure

Source: https://github.com/wkcn/tinyclip/blob/main/docs/PREPARATION.md

Directory structure for ImageNet-1k stored as tar archives.

```text
ImageNet/
├── train.tar
└── val.tar
```

--------------------------------

### ImageNet-1k individual file structure

Source: https://github.com/wkcn/tinyclip/blob/main/docs/PREPARATION.md

Directory structure for ImageNet-1k stored as individual image files.

```text
ImageNet/
├── train
│   ├── n01440764
│   │   ├── n01440764_10026.JPEG
│   │   ├── n01440764_10027.JPEG
...
├── val
│   ├── n01440764
│   │   ├── ILSVRC2012_val_00000293.JPEG
```

--------------------------------

### BibTeX citation for TinyCLIP paper

Source: https://github.com/wkcn/tinyclip/blob/main/README.md

Use this BibTeX entry to cite the TinyCLIP paper when it is helpful for your work. It includes the paper title, authors, and publication details from the IEEE/CVF International Conference on Computer Vision (ICCV) 2023.

```bibtex
@InProceedings{tinyclip,
    title     = {TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance},
    author    = {Wu, Kan and Peng, Houwen and Zhou, Zhenghong and Xiao, Bin and Liu, Mengchen and Yuan, Lu and Xuan, Hong and Valenzuela, Michael and Chen, Xi (Stephen) and Wang, Xinggang and Chao, Hongyang and Hu, Han},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {21970-21980}
}
```
Tokens:
Raw