DiTraj

https://github.com/xduzhangjiayu/ditraj
Admin
DiTraj is the first training-free trajectory control framework for DiT-based video generation...
Tokens:987
Snippets:15
Trust Score:6.3
Update:3 months ago
Show doc for...
Tokens:
Raw
### Set up DiTraj Environment

Source: https://github.com/xduzhangjiayu/ditraj/blob/main/README.md

Create and activate a Conda environment, install dependencies, and set up the diffusers library.

```bash
conda create --name DiTraj python=3.11
conda activate DiTraj
pip install -r requirements.txt
git clone --branch v0.33.1 https://github.com/huggingface/diffusers.git
cd diffusers
pip install -e .
```

--------------------------------

### Run Video Generation

Source: https://github.com/xduzhangjiayu/ditraj/blob/main/README.md

Execute the main script to generate the video with the defined trajectory. The output video will be saved as demo/output.mp4 and demo/output_box.mp4.

```bash
python run.py
```

--------------------------------

### Process Prompts for Video Generation

Source: https://github.com/xduzhangjiayu/ditraj/blob/main/README.md

Run prompt processing scripts to generate refined prompts for video generation. The output is saved to demo/test_prompts_refined.json.

```bash
python prompt_extend.py (optional)
python prompt_refine.py
```

--------------------------------

### Replace Transformer File

Source: https://github.com/xduzhangjiayu/ditraj/blob/main/README.md

Replace the local transformer_wan.py file with the one from the cloned diffusers repository.

```bash
Replace the ./module/transformer_wan.py file in the ./diffusers/src/diffusers/models/transformers/transformer_wan.py
```

--------------------------------

### Clone DiTraj Repository

Source: https://github.com/xduzhangjiayu/ditraj/blob/main/README.md

Clone the DiTraj project repository from GitHub.

```bash
git clone https://github.com/xduzhangjiayu/DiTraj.git
```

--------------------------------

### Define Simple Trajectory Bounding Boxes

Source: https://github.com/xduzhangjiayu/ditraj/blob/main/README.md

Define bounding boxes for a simple object trajectory across keyframes. Each entry is [frame_id, y1, y2, x1, x2].

```python
bboxs = [
            [0, 0.3, 0.7, 0.1, 0.4], # frame 0: Left side
            [80, 0.3, 0.7, 0.7,1.0]  # frame 80: Right side
        ]
```

--------------------------------

### Define Complex Trajectory Bounding Boxes

Source: https://github.com/xduzhangjiayu/ditraj/blob/main/README.md

Define bounding boxes for a complex object trajectory involving multiple keyframes and movements.

```python
bboxs = [
            [0, 0.05, 0.55, 0.05, 0.45], # frame 0: Top-left
            [20, 0.05, 0.55, 0.55, 0.95], # frame 20: Top-right
            [40, 0.45, 0.95, 0.55, 0.95], # frame 40: Bottom-left
            [60, 0.45, 0.95, 0.05, 0.45], # frame 60: Bottom-right
            [80, 0.05, 0.55, 0.05, 0.45], # frame 80: Top-left 
        ]
```
Tokens:
Raw