ViBT

https://github.com/yuanshi9815/vibt
Admin
ViBT (Vision Bridge Transformer at Scale) is a project that utilizes a bridge formulation for...
Tokens:3,399
Snippets:102
Trust Score:7.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Setup Conda Environment

Source: https://github.com/yuanshi9815/vibt/blob/main/README.md

Create and activate a new Conda environment for ViBT. Requires Python 3.12.

```bash
conda create -n ViBT python=3.12
conda activate ViBT
```

--------------------------------

### Import necessary libraries for ViBT video interpolation

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_frame_interpolation.ipynb

Imports required PyTorch, Diffusers, and ViBT modules for video processing and frame interpolation. Ensure all dependencies are installed.

```python
import torch
from diffusers import WanPipeline
from diffusers.utils import export_to_video, load_video
from vibt.wan import load_vibt_weight, encode_video
from vibt.scheduler import ViBTScheduler
```

--------------------------------

### Install ViBT Requirements

Source: https://github.com/yuanshi9815/vibt/blob/main/README.md

Install the ViBT package and its dependencies in the activated Conda environment. This command installs the package in editable mode.

```bash
pip install -e .
```

--------------------------------

### Prepare Source Image and Prompt

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/image_stylization.ipynb

Loads and resizes a source image, and defines a text prompt for image stylization.

```python
source_image = Image.open("../assets/000189825.jpg").resize((1024, 1024))
prompt = "Make this image into winter."
```

--------------------------------

### Load recommended styles

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_stylization.ipynb

Reads a list of recommended styles from a text file. Each line in the file represents a distinct style prompt.

```python
recommended_styles = []

with open("../assets/recommended_styles/video_stylization.txt", "r") as f:
    recommended_styles = f.read().splitlines()
```

--------------------------------

### Load Recommended Styles

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/image_stylization.ipynb

Reads recommended style prompts from a text file, filtering out empty lines.

```python
recommended_styles = []

with open("../assets/recommended_styles/image_stylization.txt", "r") as f:
    recommended_styles = [style for style in f.read().splitlines() if style.strip()]
```

--------------------------------

### Load video, prepare latents, and perform frame interpolation

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_frame_interpolation.ipynb

Loads the source video, creates an unsampled version, encodes it into latents, and then uses the configured pipeline to interpolate frames. The scheduler parameters like noise_scale and shift_gamma are set for optimal interpolation quality. The output is exported as a 60fps video.

```python
source_video = load_video("../assets/video_00000106.mp4")
unsampled_video = [source_video[(i + 3) // 4] for i in range(len(source_video) * 4 - 3)]
source_latents = encode_video(pipe, unsampled_video)
prompt = ""


pipe.scheduler.set_parameters(noise_scale=1.0, shift_gamma=5.0, seed=42)

output = pipe(
    prompt=prompt,
    num_inference_steps=20,
    guidance_scale=1.0,
    latents=source_latents,
).frames[0]
export_to_video(output, "video_60fps.mp4", fps=60)
```

--------------------------------

### Initialize Qwen Image Edit Pipeline

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/image_stylization.ipynb

Loads the pre-trained Qwen-Image-Edit model and moves it to the CUDA device for GPU acceleration.

```python
base_model_id = "Qwen/Qwen-Image-Edit"
pipe = QwenImageEditPipeline.from_pretrained(
    base_model_id, torch_dtype=torch.bfloat16
).to("cuda")
```

--------------------------------

### Load the pre-trained WanPipeline for video generation

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_frame_interpolation.ipynb

Loads the Wan-AI/Wan2.1-T2V-1.3B-Diffusers model and moves it to the CUDA-enabled GPU for faster processing. Uses bfloat16 for reduced memory usage.

```python
base_model_id = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
pipe = WanPipeline.from_pretrained(base_model_id, torch_dtype=torch.bfloat16).to("cuda")
```

--------------------------------

### Generate Stylized Image with Recommended Style

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/image_stylization.ipynb

Sets scheduler parameters and generates a stylized image using the first prompt from the loaded recommended styles. This version uses 64 inference steps.

```python
pipe.scheduler.set_parameters(noise_scale=1.0, shift_gamma=5.0, seed=42)

output = qwen_bridge_gen(
    pipe,
    prompt=recommended_styles[0],
    negative_prompt="",
    true_cfg_scale=2.0,
    num_inference_steps=64,
    image=source_image,
).images[0]

output
```

--------------------------------

### Apply ViBT weights and configure scheduler

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_colorization.ipynb

Loads specific ViBT weights for video colorization into the pipeline's transformer and replaces the default scheduler with ViBTScheduler. This step is crucial for enabling the colorization functionality.

```python
load_vibt_weight(
    pipe.transformer,
    "Yuanshi/ViBT",
    "video/video_colorization.safetensors",
)
pipe.scheduler = ViBTScheduler.from_scheduler(pipe.scheduler)
```

--------------------------------

### Generate Stylized Image with Basic Prompt

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/image_stylization.ipynb

Sets scheduler parameters and generates a stylized image using a basic prompt. The output is the first image from the generated list.

```python
pipe.scheduler.set_parameters(noise_scale=1.0, shift_gamma=5.0, seed=42)

output = qwen_bridge_gen(
    pipe,
    prompt="Change the season to winter.",
    negative_prompt="",
    true_cfg_scale=2.0,
    num_inference_steps=32,
    image=source_image,
).images[0]

output
```

--------------------------------

### Perform video colorization

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_colorization.ipynb

Loads a source video, converts it to grayscale, encodes it into latents, and then uses the configured pipeline to generate a colorized version. The output video is saved with specified FPS.

```python
source_video = load_video("../assets/video_00000106.mp4")
gray_video = [each.convert("L").convert("RGB") for each in source_video]
source_latents = encode_video(pipe, gray_video)
prompt = ""


pipe.scheduler.set_parameters(noise_scale=1.0, shift_gamma=5.0, seed=42)

output = pipe(
    prompt=prompt,
    num_inference_steps=20,
    guidance_scale=1.0,
    latents=source_latents,
).frames[0]
export_to_video(output, "video_106_coloring.mp4", fps=15)
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/image_stylization.ipynb

Imports required libraries for image editing, including PyTorch, Diffusers, PIL, and custom ViBT components.

```python
import torch
from diffusers import QwenImageEditPipeline
from PIL import Image

from vibt.scheduler import ViBTScheduler
from vibt.qwen_image import qwen_bridge_gen
```

--------------------------------

### Generate stylized video 2

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_stylization.ipynb

Loads a different source video, encodes it, applies the same stylization parameters but with a different prompt, and exports the resulting stylized video.

```python
source_video = load_video("../assets/video_00001275.mp4")
source_latents = encode_video(pipe, source_video)
prompt = recommended_styles[1]


pipe.scheduler.set_parameters(noise_scale=1.0, shift_gamma=5.0, seed=42)

output = pipe(
    prompt=prompt,
    num_inference_steps=28,
    guidance_scale=1.5,
    latents=source_latents,
).frames[0]
export_to_video(output, "output2.mp4", fps=15)
```

--------------------------------

### Load ViBT weights and set scheduler

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_stylization.ipynb

Applies custom ViBT weights to the transformer and replaces the default scheduler with ViBTScheduler for enhanced control over the stylization process.

```python
load_vibt_weight(
    pipe.transformer,
    "Yuanshi/ViBT",
    "video/video_stylization.safetensors",
)
pipe.scheduler = ViBTScheduler.from_scheduler(pipe.scheduler)
```

--------------------------------

### Generate stylized video 1

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_stylization.ipynb

Loads a source video, encodes it into latents, sets stylization parameters, and generates a stylized output video. The output video is then exported.

```python
source_video = load_video("../assets/video_00000000.mp4")
source_latents = encode_video(pipe, source_video)
prompt = recommended_styles[0]


pipe.scheduler.set_parameters(noise_scale=1.0, shift_gamma=5.0, seed=42)

output = pipe(
    prompt=prompt,
    num_inference_steps=28,
    guidance_scale=1.5,
    latents=source_latents,
).frames[0]
export_to_video(output, "output1.mp4", fps=15)
```

--------------------------------

### Load ViBT LoRA Weights and Scheduler

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/image_stylization.ipynb

Unloads existing LoRA weights, loads the ViBT LoRA weights, and sets the ViBTScheduler for the pipeline.

```python
pipe.unload_lora_weights()
pipe.load_lora_weights(
    "Yuanshi/ViBT",
    weight_name="image/image_editing.safetensors",
)

pipe.scheduler = ViBTScheduler()
```

--------------------------------

### Apply ViBT weights and configure the scheduler

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/video_frame_interpolation.ipynb

Applies the specific ViBT weights for frame interpolation to the pipeline's transformer and replaces the default scheduler with ViBTScheduler. This step is crucial for enabling the frame interpolation functionality.

```python
load_vibt_weight(
    pipe.transformer,
    "Yuanshi/ViBT",
    "video/frame_interpolation.safetensors",
)
pipe.scheduler = ViBTScheduler.from_scheduler(pipe.scheduler)
```

--------------------------------

### Load Autoreload Extensions

Source: https://github.com/yuanshi9815/vibt/blob/main/examples/image_stylization.ipynb

Enables autoreloading of modules to reflect changes during development.

```python
# autoreload
%load_ext autoreload
%autoreload 2
```

--------------------------------

### ViBT BibTeX Entry

Source: https://github.com/yuanshi9815/vibt/blob/main/README.md

BibTeX citation for the ViBT paper. Use this to cite the project in academic work.

```bibtex
@article{tan2025vision,
  title={Vision Bridge Transformer at Scale},
  author={Tan, Zhenxiong and Wang, Zeqing and Yang, Xingyi and Liu, Songhua and Wang, Xinchao},
  journal={arXiv preprint arXiv:2511.23199},
  year={2025}
}
```
Tokens:
Raw