CatV2TON

https://github.com/zheng-chong/catv2ton
Admin
CatV2TON is a DiT-based method for vision-based virtual try-on using temporal concatenation of video...
Tokens:4,366
Snippets:61
Trust Score:8.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Setup Docker Environment for EasyAnimate

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Commands to pull the EasyAnimate Docker image, run it with GPU access, clone the repository, and navigate to the video captioning directory.

```shell
# pull image
docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:
asyanimate

# enter image
docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:asyanimate

# clone code
git clone https://github.com/aigc-apps/EasyAnimate.git

# enter video_caption
cd EasyAnimate/easyanimate/video_caption
```

--------------------------------

### Train VAE Script

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/vae/README.md

Shell command to start the VAE training process. Ensure configuration parameters in 'easyanimate/vae/configs/autoencoder' are set correctly before execution.

```shell
sh scripts/train_vae.sh
```

--------------------------------

### Dataset JSON Structure (Absolute Paths)

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/vae/README.md

Example of a JSON file defining the dataset with absolute file paths. If using absolute paths, the 'data_root' configuration parameter should be omitted.

```json
[
    {
      "file_path": "/mnt/data/videos/00000001.mp4",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "video"
    },
    {
      "file_path": "/mnt/data/train/00000001.jpg",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "image"
    },
    .....
]
```

--------------------------------

### Dataset JSON Structure (Relative Paths)

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/vae/README.md

Example of a JSON file defining the dataset with relative file paths. Ensure 'file_path' correctly points to your video or image files within the dataset root.

```json
[
    {
      "file_path": "videos/00000001.mp4",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "video"
    },
    {
      "file_path": "train/00000001.jpg",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "image"
    },
    .....
]
```

--------------------------------

### Example JSON Data Format for VAE Training

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/vae/README_zh-CN.md

This JSON structure defines the input data for VAE training. 'file_path' can be relative or absolute, and 'type' specifies whether the entry is a 'video' or 'image'.

```json
[
    {
      "file_path": "videos/00000001.mp4",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "video"
    },
    {
      "file_path": "train/00000001.jpg",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "image"
    },
    .....
]
```

```json
[
    {
      "file_path": "/mnt/data/videos/00000001.mp4",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "video"
    },
    {
      "file_path": "/mnt/data/train/00000001.jpg",
      "text": "A group of young men in suits and sunglasses are walking down a city street.",
      "type": "image"
    },
    .....
]
```

--------------------------------

### Prepare Original Prompts for Beautification

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Format user-uploaded prompts into a JSONL file for the Beautiful Prompt feature. Each entry should have a 'prompt' key.

```json
{"prompt": "A stylish woman in a black leather jacket, red dress, and boots walks confidently down a damp Tokyo street."}
{"prompt": "An underwater world with realistic fish and other creatures of the sea."}
{"prompt": "a monarch butterfly perched on a tree trunk in the forest."}
{"prompt": "a child in a room with a bottle of wine and a lamp."}
{"prompt": "two men in suits walking down a hallway."}
```

--------------------------------

### Download VILA1.5 Model

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Download the VILA1.5 model for video recaptioning. Ensure you have sufficient GPU memory. Use HF_ENDPOINT for access issues.

```shell
# Add HF_ENDPOINT=https://hf-mirror.com before the command if you cannot access to huggingface.com
huggingface-cli download Efficient-Large-Model/VILA1.5-40b-AWQ --local-dir-use-symlinks False --local-dir /PATH/TO/VILA_MODEL
```

--------------------------------

### Evaluate Video Try-On Metrics

Source: https://github.com/zheng-chong/catv2ton/blob/main/README.md

This script evaluates video try-on models on ViViD-S-Test and VVT-Test datasets. Specify the ground truth and prediction folders containing MP4 files.

```bash
CUDA_VISIBLE_DEVICES=0 python eval_video_metrics.py \
--gt_folder YOUR_GT_FOLDER \
--pred_folder YOUR_PRED_FOLDER \
--num_workers 16 \
--paired
```

--------------------------------

### Deploy Local OpenAI Compatible Server with vLLM

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Deploy an OpenAI compatible server locally using vLLM to serve LLMs for prompt beautification. Ensure the model is downloaded first.

```shell
# Meta-Llama-3-8B-Instruct is sufficient for this task.
# Download it from https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct or https://www.modelscope.cn/models/LLM-Research/Meta-Llama-3-8B-Instruct to /path/to/your_llm

# deploy the OpenAI compatible server
python -m vllm.entrypoints.openai.api_server serve /path/to/your_llm --dtype auto --api-key "your_api_key"
```

--------------------------------

### Run Video Recaptioning Workflow

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Execute the complete video recaptioning pipeline. Set VILA_MODEL_PATH and REWRITE_MODEL_PATH environment variables.

```shell
VILA_MODEL_PATH=/PATH/TO/VILA_MODEL REWRITE_MODEL_PATH=/PATH/TO/REWRITE_MODEL sh scripts/stage_3_video_recaptioning.sh
```

--------------------------------

### Run Video Try-On Inference

Source: https://github.com/zheng-chong/catv2ton/blob/main/README.md

Execute the inference script for video try-on. Ensure to replace placeholders like YOUR_DATASET_PATH and OUTPUT_DIR_TO_SAVE_RESULTS with your actual paths. This command supports different datasets and various inference parameters.

```bash
CUDA_VISIBLE_DEVICES=0 python eval_video_try_on.py \
--dataset vivid | vvt \
--data_root_path YOUR_DATASET_PATH \
--output_dir OUTPUT_DIR_TO_SAVE_RESULTS \
--dataloader_num_workers 8 \
--batch_size 8 \
--seed 42 \
--mixed_precision bf16 \
--allow_tf32 \
--repaint \
--eval_pair
```

--------------------------------

### Perform Beautiful Prompt using Local vLLM Server

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Run the beautiful_prompt.py script to perform prompt beautification using a locally deployed vLLM OpenAI compatible server. Specify model, prompt, base URL, and API key.

```shell
python -m beautiful_prompt.py \
    --model /path/to/your_llm \
    --prompt "your_prompt" \
    --base_url "http://localhost:8000/v1" \
    --api_key "your_api_key"
```

--------------------------------

### Evaluate Image Try-On Metrics

Source: https://github.com/zheng-chong/catv2ton/blob/main/README.md

Use this script to evaluate the performance of image try-on models on VITONHD and DressCode datasets. Ensure ground truth and prediction folders are correctly specified.

```bash
CUDA_VISIBLE_DEVICES=0 python eval_image_metrics.py \
--gt_folder YOUR_GT_FOLDER \
--pred_folder YOUR_PRED_FOLDER \
--batch_size 16 \
--num_workers 16 \
--paired
```

--------------------------------

### Request Beautiful Prompt via OpenAI Server

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Use an OpenAI compatible server to perform beautiful prompt inference. Set OPENAI_API_KEY and OPENAI_BASE_URL environment variables.

```shell
OPENAI_API_KEY="your_openai_api_key" OPENAI_BASE_URL="your_openai_base_url" python beautiful_prompt.py \
    --model "your_model_name" \
    --prompt "your_prompt"
```

--------------------------------

### Download LLM for Rewriting Captions

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Download a local LLM, such as Meta-Llama-3-8B-Instruct, for rewriting video recaption results. Use HF_ENDPOINT for access issues.

```shell
# Add HF_ENDPOINT=https://hf-mirror.com before the command if you cannot access to huggingface.com
huggingface-cli download NousResearch/Meta-Llama-3-8B-Instruct --local-dir-use-symlinks False --local-dir /PATH/TO/REWRITE_MODEL
```

--------------------------------

### Inference for Image Try-On

Source: https://github.com/zheng-chong/catv2ton/blob/main/README.md

Perform inference for image try-on tasks on VITONHD and DressCode datasets. This script supports mixed precision and TF32 for potentially faster inference. Ensure dataset paths and output directories are correctly set.

```bash
CUDA_VISIBLE_DEVICES=0 python eval_image_try_on.py \
--dataset vitonhd | dresscode \
--data_root_path YOUR_DATASET_PATH \
--output_dir OUTPUT_DIR_TO_SAVE_RESULTS \
--dataloader_num_workers 8 \
--batch_size 8 \
--seed 42 \
--mixed_precision bf16 \
--allow_tf32 \
--repaint \
--eval_pair
```

--------------------------------

### Perform Batched Inference for Beautiful Prompt

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Run the caption_rewrite.py script for batched inference to beautify prompts. Specify paths to metadata, model, and prompt files, along with batch size and prefix.

```shell
# Meta-Llama-3-8B-Instruct is sufficient for this task.
# Download it from https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct or https://www.modelscope.cn/models/LLM-Research/Meta-Llama-3-8B-Instruct to /path/to/your_llm

python caption_rewrite.py \
    --video_metadata_path datasets/original_prompt.jsonl \
    --caption_column "prompt" \
    --batch_size 1 \
    --model_name /path/to/your_llm \
    --prompt prompt/beautiful_prompt.txt \
    --prefix '"detailed description": ' \
    --saved_path datasets/beautiful_prompt.jsonl \
    --saved_freq 1
```

--------------------------------

### Video Splitting Script

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Execute the shell script for video splitting. This process uses PySceneDetect and FFmpeg to segment videos into clips, discarding clips shorter than 3 seconds and recursively splitting longer ones.

```shell
sh scripts/stage_1_video_splitting.sh
```

--------------------------------

### Video Filtering Script

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Run the script for video filtering, which includes aesthetic, text, and motion filtering. This step helps select high-quality videos for recaptioning. Ensure the correct model is accessible for aesthetic score computation.

```shell
sh scripts/stage_2_video_filtering.sh
```

--------------------------------

### Video Filtering Script with Hugging Face Mirror

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/README.md

Alternative command for the video filtering script, using a Hugging Face mirror endpoint to resolve potential access issues with the 'google/siglip-so400m-patch14-384' model.

```shell
HF_ENDPOINT=https://hf-mirror.com sh scripts/stage_2_video_filtering.sh
```

--------------------------------

### Modify dtype Property in Long-CLIP Model

Source: https://github.com/zheng-chong/catv2ton/blob/main/easyanimate/video_caption/utils/longclip/README.md

This modification adjusts the dtype property to correctly handle VideoCLIP-XL inference by checking for the 'visual' attribute before accessing its components, or falling back to 'token_embedding' if 'visual' is not present.

```python
def dtype(self):
    # Fix: the VideoCLIP-XL inference.
    if hasattr(self, "visual"):
        return self.visual.conv1.weight.dtype
    else:
        return self.token_embedding.weight.dtype
```
Tokens:
Raw