FastVideo

https://haoailab.com/FastVideo
Admin
FastVideo is a unified inference and post-training framework for accelerated video generation with...
Tokens:1,251,122
Snippets:9,247
Trust Score:5.1
Update:5 months ago
Show doc for...
Tokens:
Raw
### Verify FastVideo Setup

Source: https://haoailab.com/FastVideo/contributing/developer_env/runpod?q=

Run a simple Python import check and execute the test suite to verify that FastVideo is installed and configured correctly.

```bash
cd /FastVideo
python -c "import fastvideo; print('OK')"
pytest tests/ -q --no-header
```

--------------------------------

### Command-line Conversion (with torchrun)

Source: https://haoailab.com/FastVideo/api/fastvideo/train/entrypoint/dcp_to_diffusers

Alternatively, use torchrun for conversion, which is also supported. This example shows a basic setup with one process per node.

```bash
torchrun --nproc_per_node=1         -m fastvideo.train.entrypoint.dcp_to_diffusers         --checkpoint ... --output-dir ...
```

--------------------------------

### Run Basic Inference Example

Source: https://haoailab.com/FastVideo/inference/examples/basic?q=

Execute the basic inference script to generate a video using default settings. This is the starting point for new users.

```bash
python examples/inference/basic/basic.py
```

--------------------------------

### YAML Configuration Example

Source: https://haoailab.com/FastVideo/design/training_architecture

Example YAML configuration file demonstrating how to define models, training methods, and training parameters for FastVideo.

```yaml
models:
  student:
    _target_: fastvideo.train.models.wan.WanModel
    init_from: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
    trainable: true
  teacher:
    _target_: fastvideo.train.models.wan.WanModel
    init_from: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
    trainable: false
  critic:
    _target_: fastvideo.train.models.wan.WanModel
    init_from: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
    trainable: true

method:
  _target_: fastvideo.train.methods.distribution_matching.dmd2.DMD2Method
  rollout_mode: simulate
  dmd_denoising_steps: [1000, 850, 700, 550, 350, 275, 200, 125]
  generator_update_interval: 5
  real_score_guidance_scale: 3.5
  # ...

training:
  distributed: { num_gpus: 8, sp_size: 1, tp_size: 1 }
  data: { data_path: ..., num_latent_t: 20, num_frames: 77 }
  optimizer: { learning_rate: 2.0e-6, betas: [0.0, 0.999] }
  loop: { max_train_steps: 4000 }
  checkpoint: { output_dir: outputs/my_run }

callbacks:
  grad_clip: { max_grad_norm: 1.0 }
  validation: { pipeline_target: ..., every_steps: 100 }


```

--------------------------------

### Install FastVideo from Source with uv

Source: https://haoailab.com/FastVideo/getting_started/installation/gpu

Install FastVideo in editable mode using uv after cloning the repository.

```bash
uv pip install -e .

```

--------------------------------

### Install and Run Pre-commit Hooks Locally

Source: https://haoailab.com/FastVideo/contributing/pull_requests?q=

Install pre-commit using uv and then install the hooks. Run all checks on all files to reproduce and auto-fix issues locally before pushing.

```bash
# Install pre-commit if needed
uv pip install pre-commit
pre-commit install

# Run all checks on all files
pre-commit run --all-files

```

--------------------------------

### Install Sage Attention from Source

Source: https://haoailab.com/FastVideo/inference/optimizations

Compile and install Sage Attention from its source repository. Use 'pip install -e .' as an alternative if preferred.

```bash
git clone https://github.com/thu-ml/SageAttention.git
cd sageattention
python setup.py install  # or pip install -e .

```

--------------------------------

### FastVideo Gradio Interface Setup

Source: https://haoailab.com/FastVideo/inference/examples/local?q=

Configures and launches the FastVideo Gradio interface. It parses command-line arguments for model paths and server settings, loads models, and starts the web server.

```python
import argparse
import os

def main():
    parser = argparse.ArgumentParser(description="FastVideo Gradio Local Demo")
    parser.add_argument("--t2v_model_paths", type=str,
                        default="FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
                        help="Comma separated list of paths to the T2V model(s)")
    parser.add_argument("--host", type=str, default="0.0.0.0",
                        help="Host to bind to")
    parser.add_argument("--port", type=int, default=7860,
                        help="Port to bind to")
    args = parser.parse_args()
    generators = {}
    default_params = {}
    model_paths = args.t2v_model_paths.split(",")
    for model_path in model_paths:
        print(f"Loading model: {model_path}")
        setup_model_environment(model_path)
        generators[model_path] = VideoGenerator.from_pretrained(model_path)
        default_params[model_path] = SamplingParam.from_pretrained(model_path)
    demo = create_gradio_interface(default_params, generators)
    print(f"Starting Gradio frontend at http://{args.host}:{args.port}")
    print(f"T2V Models: {args.t2v_model_paths}")

    from fastapi import FastAPI, Request, HTTPException
    from fastapi.responses import HTMLResponse, FileResponse
    import uvicorn

    app = FastAPI()

    @app.get("/logo.png")
    def get_logo():
        return FileResponse(
            "assets/full.svg",
            media_type="image/svg+xml",
            headers={
                "Cache-Control": "public, max-age=3600",
                "Access-Control-Allow-Origin": "*"
            }
        )

    @app.get("/favicon.ico")
    def get_favicon():
        favicon_path = "assets/icon-simple.svg"

        if os.path.exists(favicon_path):
            return FileResponse(
                favicon_path, 
                media_type="image/svg+xml",
                headers={
                    "Cache-Control": "public, max-age=3600",
                    "Access-Control-Allow-Origin": "*"
                }
            )
        else:
            raise HTTPException(status_code=404, detail="Favicon not found")

    @app.get("/", response_class=HTMLResponse)
    def index(request: Request):
        base_url = str(request.base_url).rstrip('/')
        return f"""
        <!DOCTYPE html>
        <html lang="en">
        <head>
            <meta charset="UTF-8" />
            <meta name="viewport" content="width=device-width, initial-scale=1.0" />

            <title>FastWan</title>
            <meta name="title" content="FastWan">
            <meta name="description" content="Make video generation go blurrrrrrr">
            <meta name="keywords" content="FastVideo, video generation, AI, machine learning, FastWan">

            <meta property="og:type" content="website">
            <meta property="og:url" content="{base_url}/">
            <meta property="og:title" content="FastWan">
            <meta property="og:description" content="Make video generation go blurrrrrrr">
            <meta property="og:image" content="{base_url}/logo.png">
            <meta property="og:image:width" content="1200">
            <meta property="og:image:height" content="630">
            <meta property="og:site_name" content="FastWan">

            <meta property="twitter:card" content="summary_large_image">
            <meta property="twitter:url" content="{base_url}/">
            <meta property="twitter:title" content="FastWan">
            <meta property="twitter:description" content="Make video generation go blurrrrrrr">
            <meta property="twitter:image" content="{base_url}/logo.png">
            <link rel="icon" type="image/png" sizes="32x32" href="/favicon.ico">
            <link rel="icon" type="image/png" sizes="16x16" href="/favicon.ico">
            <link rel="apple-touch-icon" href="/favicon.ico">
            <style>
                body, html {{
                    margin: 0;
                    padding: 0;
                    height: 100%;
                    overflow: hidden;
                }}
                iframe {{ 
                    width: 100%;

```

--------------------------------

### Start Game Function

Source: https://haoailab.com/FastVideo/inference/examples/local?q=

Initializes and starts the game, setting up the generator and initial state.

```python
def start_game(model_name, seed_val, randomize, state):
            if state.get("initialized"):
                return state, state.get("seed", 0), "Block: 0 / 50", None, gr.update(value="Start"), gr.update(interactive=False)

            config = VARIANT_CONFIG.get(model_name)
            model_path = config["model_path"]

            if not generators.get(model_path):
                print(f"Loading model: {model_path}")
                setup_model_environment(model_path)
                generators[model_path] = StreamingVideoGenerator.from_pretrained(
                    model_path,
                    num_gpus=1,
                    use_fsdp_inference=True,
                    dit_cpu_offload=True,
                    vae_cpu_offload=False,
                    text_encoder_cpu_offload=True,
                    pin_cpu_memory=True,
                )

            generator = generators.get(model_path)
            mode = state["mode"]
            frames_per_block = state["frames_per_block"]

            # Parse inputs to tensors
            action = get_action_tensors(mode, keyboard_key, mouse_key)
            keyboard_cond, mouse_cond = expand_action_to_frames(action, frames_per_block)

            # run step async
            # inference_start_time = time.time()
            frames, block_future = await generator.step_async(keyboard_cond, mouse_cond)
            # inference_time = time.time() - inference_start_time

            # wait for block file to be written
            block_path = await asyncio.to_thread(block_future.result) if block_future else None
            state["block_idx"] = generator.block_idx
            block_str = f"Block: {state['block_idx']} / {state['max_blocks']}"

            # total_time = time.time() - total_start_time

            # Timing breakdown
            # timing_html = create_timing_display(inference_time, total_time, [], frames_per_block)

            return state, state.get("seed", 0), block_str, block_path, gr.update(), gr.update()

```

--------------------------------

### Install FastVideo from source

Source: https://haoailab.com/FastVideo/getting_started/installation

Install FastVideo directly from its GitHub repository. This includes cloning the repository, navigating into the directory, and installing the package in editable mode. Optionally, flash-attn can also be installed.

```bash
git clone https://github.com/hao-ai-lab/FastVideo.git
cd FastVideo
uv pip install -e .

# optional: install flash-attn
uv pip install flash-attn --no-build-isolation -v

```

```bash
pip install -e .
pip install flash-attn --no-build-isolation -v

```

--------------------------------

### Install FastVideo with uv

Source: https://haoailab.com/FastVideo/getting_started/installation/gpu

Install the FastVideo package using uv in the activated environment.

```bash
uv pip install fastvideo

```

--------------------------------

### Install FastVideo using uv

Source: https://haoailab.com/FastVideo/getting_started/installation

Use uv for faster and more stable installations. This snippet creates and activates a new uv environment and then installs FastVideo.

```bash
# Create and activate a new uv environment
uv venv --python 3.12 --seed
source .venv/bin/activate

uv pip install fastvideo

```

--------------------------------

### Install and Run Pre-commit Hooks Locally

Source: https://haoailab.com/FastVideo/contributing/pull_requests

Install pre-commit and run all checks locally to reproduce and auto-fix issues before pushing.

```bash
# Install pre-commit if needed
uv pip install pre-commit
pre-commit install

# Run all checks on all files
pre-commit run --all-files
```

--------------------------------

### GEN3C 3D-aware Camera-Controlled Video Generation

Source: https://haoailab.com/FastVideo/inference/examples/basic?q=

This example demonstrates 3D-aware video generation using the GEN3C model with camera control. It requires specific setup including installing MoGe, downloading and converting model weights, and providing an input image.

```python
"""
GEN3C: 3D-aware camera-controlled video generation.

This example generates a video from a single input image with camera control.
The pipeline uses MoGe depth estimation, 3D point cloud forward warping,
and the GEN3C diffusion model.

Requirements:
  1. Install MoGe:
     pip install git+https://github.com/microsoft/MoGe.git
     If you hit `ImportError: libGL.so.1`, install:
     sudo apt-get update && sudo apt-get install -y libgl1 libglib2.0-0 libsm6 libxext6 libxrender1
  2. Download and convert weights:
     huggingface-cli download nvidia/GEN3C-Cosmos-7B --local-dir official_weights/GEN3C-Cosmos-7B
     python scripts/checkpoint_conversion/convert_gen3c_to_fastvideo.py \
       --source ./official_weights/GEN3C-Cosmos-7B/model.pt \
       --output ./converted_weights/GEN3C-Cosmos-7B \
       --components-source nvidia/Cosmos-Predict2-2B-Video2World
  3. Provide an input image for 3D-conditioned generation.
"""

import argparse

from fastvideo import VideoGenerator


def main():
    parser = argparse.ArgumentParser(description="GEN3C video generation")
    parser.add_argument("--model_path",
                        type=str,
                        default="converted_weights/GEN3C-Cosmos-7B")
    parser.add_argument("--image_path",
                        type=str,
                        default=None,
                        help="Input image for 3D cache conditioning")
    parser.add_argument("--prompt",
                        type=str,
                        default="A slow camera pan over a sunlit landscape.")
    parser.add_argument(
        "--negative_prompt",
        type=str,
        default=(
            "The video captures a series of frames showing ugly scenes, static with no motion, motion blur, "
            "over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, "
            "underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, "
            "jerky movements, low frame rate, artifacting, color banding, unnatural transitions, outdated special "
            "effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, visual noise, and "
            "flickering. Overall, the video is of poor quality."
        ),
    )
    parser.add_argument("--trajectory",
                        type=str,
                        default="left",
                        choices=[
                            "left", "right", "up", "down", "zoom_in",
                            "zoom_out", "clockwise", "counterclockwise", "none"
                        ])
    parser.add_argument("--movement_distance", type=float, default=0.3)

```

--------------------------------

### Launch Training with Helper Script

Source: https://haoailab.com/FastVideo/training/train_infra?q=

Use the provided helper script to launch training. It automatically detects GPUs and sets up torchrun. Configuration is done via a YAML file.

```bash
examples/train/run.sh examples/train/distill_wan2.1_t2v_1.3B_dmd2.yaml
```

--------------------------------

### Start Training in FastVideo ModelBase

Source: https://haoailab.com/FastVideo/api/fastvideo/train?q=

This method is called once before the training loop begins. It serves as a hook for any setup required at the start of training.

```python
def on_train_start(self) -> None:
    """Called once before the training loop begins."""

```

--------------------------------

### Install FastVideo using uv

Source: https://haoailab.com/FastVideo/inference/inference_quick_start

Installs FastVideo in a new Python 3.12 environment using uv. Recommended for faster and more stable environment setup.

```bash
# Create and activate a new uv environment
uv venv --python 3.12 --seed
source .venv/bin/activate

# Install FastVideo
uv pip install fastvideo

```

--------------------------------

### Setup Model Environment

Source: https://haoailab.com/FastVideo/inference/examples/local?q=

Configures environment variables for attention backend and logging based on the model path. Use FLASH_ATTN for 'fullattn' models and VIDEO_SPARSE_ATTN otherwise. Enables stage logging.

```python
import argparse
import os
import base64
import time

import gradio as gr
from fastvideo.entrypoints.video_generator import VideoGenerator
from fastvideo.api.sampling_param import SamplingParam
from copy import deepcopy


MODEL_PATH_MAPPING = {
    "FastWan2.1-T2V-1.3B": "FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
    # "FastWan2.2-TI2V-5B-FullAttn": "FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers",
}

def create_timing_display(inference_time, total_time, stage_execution_times, num_frames):
    dit_denoising_time = f"{stage_execution_times[5]:.2f}s" if len(stage_execution_times) > 5 else "N/A"

    timing_html = f"""
    <div style="margin: 10px 0;">
        <h3 style="text-align: center; margin-bottom: 10px;">⏱️ Timing Breakdown</h3>
        <div style="display: grid; grid-template-columns: repeat(5, 1fr); gap: 10px; margin-bottom: 10px;">
            <div class="timing-card timing-card-highlight">
                <div style="font-size: 20px;">🚀</div>
                <div style="font-weight: bold; margin: 3px 0; font-size: 14px;">DiT Denoising</div>
                <div style="font-size: 16px; color: #ffa200; font-weight: bold;">{dit_denoising_time}</div>
            </div>
            <div class="timing-card">
                <div style="font-size: 20px;">🧠</div>
                <div style="font-weight: bold; margin: 3px 0; font-size: 14px;">E2E (w. vae/text encoder)</div>
                <div style="font-size: 16px; color: #2563eb;">{inference_time:.2f}s</div>
            </div>
            <div class="timing-card">
                <div style="font-size: 20px;">🎬</div>
                <div style="font-weight: bold; margin: 3px 0; font-size: 14px;">Video Encoding</div>
                <div style="font-size: 16px; color: #dc2626;">N/A</div>
            </div>
            <div class="timing-card">
                <div style="font-size: 20px;">🌐</div>
                <div style="font-weight: bold; margin: 3px 0; font-size: 14px;">Network Transfer</div>
                <div style="font-size: 16px; color: #059669;">N/A</div>
            </div>
            <div class="timing-card">
                <div style="font-size: 20px;">📊</div>
                <div style="font-weight: bold; margin: 3px 0; font-size: 14px;">Total Processing</div>
                <div style="font-size: 18px; color: #0277bd;">{total_time:.2f}s</div>
            </div>
        </div>"
```

```python
def setup_model_environment(model_path: str) -> None:
    if "fullattn" in model_path.lower():
        os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "FLASH_ATTN"
    else:
        os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "VIDEO_SPARSE_ATTN"
    os.environ["FASTVIDEO_STAGE_LOGGING"] = "1"
```

--------------------------------

### Install FastVideo with uv

Source: https://haoailab.com/FastVideo/getting_started/quick_start?q=

Installs FastVideo and its dependencies using the uv package manager. It's recommended to use uv for a faster and more stable setup.

```bash
uv venv --python 3.12 --seed
source .venv/bin/activate

# Install FastVideo
uv pip install fastvideo

```

--------------------------------

### Download and Preprocess Dataset

Source: https://haoailab.com/FastVideo/training/data_preprocess?q=

Download a sample dataset and run the preprocessing script. Ensure you have the necessary scripts and data paths configured.

```bash
# Download the crush-smol dataset
python scripts/huggingface/download_hf.py \
    --repo_id "wlsaidhi/crush-smol-merged" \
    --local_dir "data/crush-smol" \
    --repo_type "dataset"

# Run preprocessing
bash examples/training/finetune/wan_t2v_1.3B/crush_smol/preprocess_wan_data_t2v_new.sh

```

--------------------------------

### on_train_start

Source: https://haoailab.com/FastVideo/api/fastvideo/train?q=

A hook called once before the training loop begins. Useful for any setup required at the start of training.

```APIDOC
## on_train_start

### Description
Called once before the training loop begins.

### Method
```python
def on_train_start(self) -> None:
    """Called once before the training loop begins."""
```
```

--------------------------------

### RunPod Custom Template Container Start Command

Source: https://haoailab.com/FastVideo/contributing/developer_env/runpod

This command is used as the 'Container Start Command' when creating a custom RunPod template. It installs OpenSSH server and configures it to allow key-based authentication.

```bash
bash -c "apt update;DEBIAN_FRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $_;chmod 700 ~/.ssh;echo \"$PUBLIC_KEY\" >> authorized_keys;chmod 700 authorized_keys;service ssh start;sleep infinity"

```

--------------------------------

### Run FastVideo Gradio Local Demo

Source: https://haoailab.com/FastVideo/inference/examples/local

Execute this command to start the Gradio web server for the FastVideo demo. The interface will be accessible at http://0.0.0.0:7860.

```bash
python examples/inference/gradio/local/gradio_local_demo.py

```

--------------------------------

### Custom Template Container Start Command

Source: https://haoailab.com/FastVideo/contributing/developer_env/runpod?q=

This command is used as the Container Start Command when setting up a custom RunPod template from scratch. It installs OpenSSH server, configures SSH access with your public key, and keeps the container running.

```bash
bash -c "apt update;DEBIAN_FRONTEND=noninteractive apt-get install openssh-server -y;mkdir -p ~/.ssh;cd $_;chmod 700 ~/.ssh;echo \"$PUBLIC_KEY\" >> authorized_keys;chmod 700 authorized_keys;service ssh start;sleep infinity"
```

--------------------------------

### Run FastVideo Benchmark

Source: https://haoailab.com/FastVideo/api/fastvideo/entrypoints/cli/bench

Example of how to run a benchmark against a FastVideo server. Specify the dataset, number of prompts, and the server port.

```bash
fastvideo bench --dataset vbench --num-prompts 20 --port 8000
```

--------------------------------

### Basic MatrixGame Streaming Example

Source: https://haoailab.com/FastVideo/inference/examples/basic?q=

This Python script demonstrates how to set up a basic streaming video generator for the MatrixGame. It imports necessary components from FastVideo and sets up the model variant. This example is a starting point for real-time video generation.

```python
from fastvideo.entrypoints.streaming_generator import StreamingVideoGenerator
from fastvideo.models.dits.matrixgame.utils import get_current_action_async, expand_action_to_frames

import torch
import asyncio

# Available variants: "base_distilled_model", "gta_distilled_model", "templerun_distilled_model"
# Each variant has different keyboard_dim:
#   - base_distilled_model: keyboard_dim=4
#   - gta_distilled_model: keyboard_dim=2
#   - templerun_distilled_model: keyboard_dim=7 (keyboard only, no mouse)
MODEL_VARIANT = "base_distilled_model"

```

--------------------------------

### Launch Training with Helper Script

Source: https://haoailab.com/FastVideo/training/train_infra

Use the `run.sh` helper script to launch training. It auto-detects GPUs and sets up `torchrun`. Environment variables can override default distributed settings.

```bash
examples/train/run.sh examples/train/distill_wan2.1_t2v_1.3B_dmd2.yaml

```

```bash
NUM_GPUS=4 NNODES=2 NODE_RANK=0 \
    MASTER_ADDR=10.0.0.1 MASTER_PORT=29501 \
    bash examples/train/run.sh my_config.yaml

```

--------------------------------

### ServeSubcommand

Source: https://haoailab.com/FastVideo/api/fastvideo/entrypoints/cli

Starts an OpenAI-compatible API server.

```APIDOC
## ServeSubcommand

### Description
Starts an OpenAI-compatible API server.

### Usage
```python
ServeSubcommand()
```
```

--------------------------------

### Get PyTorch NPU Device

Source: https://haoailab.com/FastVideo/api/fastvideo/platforms/npu?q=

Use this class method to retrieve the PyTorch NPU device. Ensure PyTorch is installed with NPU support.

```python
get_torch_device()
```

```python
@classmethod
def get_torch_device(cls):
    """
    Return torch.npu
    """
    return torch.npu
```

--------------------------------

### YAML Configuration for EMA Callback

Source: https://haoailab.com/FastVideo/api/fastvideo/train/callbacks/ema

Example of how to configure the EMA callback in a YAML file. Specifies the decay rate and the iteration to start EMA updates.

```yaml
callbacks:
  ema:
    decay: 0.9999
    start_iter: 0
```

--------------------------------

### Model and Config Mapping Example

Source: https://haoailab.com/FastVideo/contributing/coding_agents?q=

Illustrates how to define model classes and parameter name mappings for aligning official checkpoints with FastVideo's structure. This is crucial for correct weight loading.

```python
# Official model (simplified) in the upstream repo.
class OfficialWanTransformer(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.patch_embedding = torch.nn.Conv3d(16, 1536, kernel_size=2, padding=0)

    def forward(self, x):
        return self.patch_embedding(x)

# FastVideo model (simplified) in fastvideo/models/dits/wanvideo.py
class PatchEmbed(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.proj = torch.nn.Conv3d(16, 1536, kernel_size=2, padding=0)

    def forward(self, x):
        return self.proj(x)

class WanTransformer3DModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.patch_embedding = PatchEmbed()

    def forward(self, x):
        return self.patch_embedding(x)

# Mapping defined in a config (simplified; see the real mapping in
# fastvideo/configs/models/dits/wanvideo.py)
param_names_mapping = {
    r"^patch_embedding\.(.*)$": r"patch_embedding.proj.\1",
    r"^blocks\.(\d+)\.attn1\.to_q\.(.*)$": r"blocks.\1.to_q.\2",
}

def apply_regex_map(state_dict, mapping):
    # Pseudocode: apply regex substitutions in order
    ...

# Official checkpoint keys (example)
official = {
    "patch_embedding.weight": ...,
    "blocks.0.attn1.to_q.weight": ..., 
}

```

--------------------------------

### SessionStore.hydrate

Source: https://haoailab.com/FastVideo/api/fastvideo/entrypoints/streaming/session_store?q=

Installs a given state as the starting point for a session. If session_id is not provided, a new UUID4 ID is generated. If session_id is provided, it overwrites any existing state for that ID.

```APIDOC
## hydrate

### Description
Install `state` as the starting point for a session.
When `session_id` is `None` the store allocates a fresh id (UUID4); when provided the store uses it verbatim, overwriting any prior state at that id.

### Method Signature
`hydrate(state: ContinuationState, *, session_id: str | None = None) -> str`

### Parameters
#### Keyword Parameters
- **state** (ContinuationState) - Required - The initial state for the session.
- **session_id** (str | None) - Optional - The ID of the session to hydrate. If None, a new session ID is generated.

### Returns
- **str** - The session ID.
```

--------------------------------

### Create uv Environment

Source: https://haoailab.com/FastVideo/getting_started/installation/gpu?q=

Creates a new Python environment using uv, recommended for faster and more stable setup. Use `--seed` to install pip and setuptools.

```bash
# (Recommended) Create a new uv environment. Use `--seed` to install `pip` and `setuptools`.
uv venv --python 3.12 --seed
source .venv/bin/activate

```

--------------------------------

### FastVideo Config File Example

Source: https://haoailab.com/FastVideo/inference/cli

An example configuration file in YAML format for FastVideo generation. This file specifies model paths, engine settings, and generation request parameters.

```yaml
generator:
  model_path: FastVideo/FastHunyuan-diffusers
  engine:
    num_gpus: 2
    parallelism:
      sp_size: 2
      tp_size: 1
request:
  prompt: A capybara lounging in a hammock
  sampling:
    num_frames: 45
    height: 720
    width: 1280
    num_inference_steps: 6
    seed: 1024
  output:
    output_path: outputs/

```

--------------------------------

### Get Basic Model Information - Python

Source: https://haoailab.com/FastVideo/api/fastvideo/entrypoints/openai

Retrieves basic information about the currently configured model. This endpoint is useful for understanding the server's current model setup.

```python
from fastapi import APIRouter

from fastvideo.state import get_server_args


router = APIRouter()


@router.get("/model_info")
async def model_info():
    """Get basic model information"""
    args = get_server_args()
    return {"model_path": args.model_path}
```

--------------------------------

### Finetune Wan2.1-T2V-1.3B Model

Source: https://haoailab.com/FastVideo/training/examples/wan_t2v_1.3B_crush_smol

This script configures and launches the finetuning process for the Wan2.1-T2V-1.3B model. It sets up environment variables, defines training, parallel, model, dataset, validation, and optimizer arguments, and then executes the training pipeline using torchrun.

```bash
#!/bin/bash

export WANDB_BASE_URL="https://api.wandb.ai"
export WANDB_MODE=online
export TOKENIZERS_PARALLELISM=false
# export FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA

MODEL_PATH="Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
DATA_DIR="data/crush-smol_processed_t2v/combined_parquet_dataset/"
VALIDATION_DATASET_FILE="$(dirname "$0")/validation.json"
NUM_GPUS=4
# export CUDA_VISIBLE_DEVICES=4,5


# Training arguments
training_args=(
  --tracker_project_name "wan_t2v_finetune"
  --output_dir "checkpoints/wan_t2v_finetune"
  --max_train_steps 5000
  --train_batch_size 1
  --train_sp_batch_size 1
  --gradient_accumulation_steps 8
  --num_latent_t 20
  --num_height 480
  --num_width 832
  --num_frames 77
  --enable_gradient_checkpointing_type "full"
)

# Parallel arguments
parallel_args=(
  --num_gpus $NUM_GPUS 
  --sp_size $NUM_GPUS 
  --tp_size 1
  --hsdp_replicate_dim 1
  --hsdp_shard_dim $NUM_GPUS
)

# Model arguments
model_args=(
  --model_path $MODEL_PATH
  --pretrained_model_name_or_path $MODEL_PATH
)

# Dataset arguments
dataset_args=(
  --data_path $DATA_DIR
  --dataloader_num_workers 1
)

# Validation arguments
validation_args=(
  --log_validation 
  --validation_dataset_file $VALIDATION_DATASET_FILE
  --validation_steps 200
  --validation_sampling_steps "50" 
  --validation_guidance_scale "3.0"
)

# Optimizer arguments
optimizer_args=(
  --learning_rate 5e-5
  --mixed_precision "bf16"
  --weight_only_checkpointing_steps 1000
  --training_state_checkpointing_steps 1000
  --weight_decay 1e-4
  --max_grad_norm 1.0
)

# Miscellaneous arguments
miscellaneous_args=(
  --inference_mode False
  --checkpoints_total_limit 3
  --training_cfg_rate 0.1
  --multi_phased_distill_schedule "4000-1"
  --not_apply_cfg_solver
  --dit_precision "fp32"
  --num_euler_timesteps 50
  --ema_start_step 0
  --enable_gradient_checkpointing_type "full"
  # --resume_from_checkpoint "checkpoints/wan_t2v_finetune/checkpoint-2500"
)

torchrun \
  --nnodes 1 \
  --nproc_per_node $NUM_GPUS \
    fastvideo/training/wan_training_pipeline.py \
    "${parallel_args[@]}" \
    "${model_args[@]}" \
    "${dataset_args[@]}" \
    "${training_args[@]}" \
    "${optimizer_args[@]}" \
    "${validation_args[@]}" \
    "${miscellaneous_args[@]}"

```

--------------------------------

### hydrate

Source: https://haoailab.com/FastVideo/api/fastvideo/entrypoints/streaming/session_store

Installs a given state as the starting point for a session. If session_id is None, a new ID is generated. If session_id is provided, it's used directly, overwriting any existing state.

```APIDOC
## hydrate

### Description
Install `state` as the starting point for a session.
When `session_id` is `None` the store allocates a fresh id (UUID4); when provided the store uses it verbatim, overwriting any prior state at that id.

### Method Signature
`hydrate(state: ContinuationState, *, session_id: str | None = None) -> str`
```

--------------------------------

### run_server

Source: https://haoailab.com/FastVideo/api/fastvideo/entrypoints/streaming/server?q=

Launches the streaming server. It boots a `fastvideo.VideoGenerator` and serves the `build_app` application via uvicorn.

```APIDOC
## run_server

### Description
Launch the streaming server. Boots a :class:`fastvideo.VideoGenerator` from ``serve_config.generator`` unless ``generator`` is provided, then serves ``build_app(...)`` via uvicorn.

### Method
```python
def run_server(serve_config: ServeConfig, *, generator: _GeneratorProto | None = None) -> None
```

### Parameters
- **serve_config** (ServeConfig) - Configuration for serving.
- **generator** (_GeneratorProto | None, optional) - The video generator. If None, it will be created from `serve_config.generator`. Defaults to None.
```

--------------------------------

### Hydrate Session State

Source: https://haoailab.com/FastVideo/api/fastvideo/entrypoints/streaming/session_store

Installs a given state as the starting point for a session. If session_id is None, a fresh ID is allocated. If provided, the store uses it verbatim, overwriting any prior state.

```python
@abstractmethod
def hydrate(
    self,
    state: ContinuationState,
    *,
    session_id: str | None = None,
) -> str:
    """Install ``state`` as the starting point for a session.

    When ``session_id`` is ``None`` the store allocates a fresh id
    (UUID4); when provided the store uses it verbatim, overwriting
    any prior state at that id.
    """

```

--------------------------------

### Main Function for Local Demo

Source: https://haoailab.com/FastVideo/inference/examples/local

Parses command-line arguments, loads T2V models, sets up the Gradio interface, and starts the web server. This is the entry point for running the local demo.

```python
import argparse
import time
import random
import gradio as gr

# Assume VideoGenerator, SamplingParam, setup_model_environment are defined elsewhere
# from fastvideo.inference.video_generator import VideoGenerator
# from fastvideo.inference.sampling_param import SamplingParam
# from fastvideo.inference.utils import setup_model_environment

def main():
    parser = argparse.ArgumentParser(description="FastVideo Gradio Local Demo")
    parser.add_argument("--t2v_model_paths", type=str,
                        default="FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
                        help="Comma separated list of paths to the T2V model(s)")
    parser.add_argument("--host", type=str, default="0.0.0.0",
                        help="Host to bind to")
    parser.add_argument("--port", type=int, default=7860,
                        help="Port to bind to")
    args = parser.parse_args()
    generators = {}
    default_params = {}
    model_paths = args.t2v_model_paths.split(",")
    for model_path in model_paths:
        print(f"Loading model: {model_path}")
        # setup_model_environment(model_path) # Placeholder for actual environment setup
        # generators[model_path] = VideoGenerator.from_pretrained(model_path) # Placeholder
        # default_params[model_path] = SamplingParam.from_pretrained(model_path) # Placeholder
        
        # Mock objects for demonstration purposes if actual classes are not available
        class MockGenerator:
            def __call__(self, *args, **kwargs):
                print("MockGenerator called")
                return ["mock_frame_1", "mock_frame_2"] # Simulate video frames
        class MockSamplingParam:
            pass
        generators[model_path] = MockGenerator()
        default_params[model_path] = MockSamplingParam()

    demo = create_gradio_interface(default_params, generators)
    print(f"Starting Gradio frontend at http://{args.host}:{args.port}")
    print(f"T2V Models: {args.t2v_model_paths}")

    # The following FastAPI and uvicorn setup is typically run in a separate script or process
    # For this example, we'll just print the Gradio launch command.
    # To run this locally, you would typically launch Gradio separately:
    # demo.launch(server_name=args.host, server_port=args.port)

    # If you were to integrate FastAPI, it might look like this:
    # from fastapi import FastAPI, Request
    # from fastapi.responses import HTMLResponse
    # import uvicorn
    # app = FastAPI()
    # @app.get("/", response_class=HTMLResponse)
    # async def read_root(request: Request):
    #     # This would serve the Gradio interface or a custom index page
    #     return "<html><body><h1>FastVideo Demo</h1></body></html>"
    # 
    # if __name__ == "__main__":
    #     uvicorn.run(app, host=args.host, port=args.port)

if __name__ == "__main__":
    main()

```

--------------------------------

### GEN3C: 3D-Aware Video Generation Setup

Source: https://haoailab.com/FastVideo/inference/examples/basic

Sets up the GEN3C pipeline for 3D-aware camera-controlled video generation. Requires installing MoGe, downloading and converting model weights, and providing an input image.

```python
"""
GEN3C: 3D-aware camera-controlled video generation.

This example generates a video from a single input image with camera control.
The pipeline uses MoGe depth estimation, 3D point cloud forward warping,
and the GEN3C diffusion model.

Requirements:
  1. Install MoGe:
     pip install git+https://github.com/microsoft/MoGe.git
     If you hit `ImportError: libGL.so.1`, install:
     sudo apt-get update && sudo apt-get install -y libgl1 libglib2.0-0 libsm6 libxext6 libxrender1
  2. Download and convert weights:
     huggingface-cli download nvidia/GEN3C-Cosmos-7B --local-dir official_weights/GEN3C-Cosmos-7B
     python scripts/checkpoint_conversion/convert_gen3c_to_fastvideo.py \
       --source ./official_weights/GEN3C-Cosmos-7B/model.pt \
       --output ./converted_weights/GEN3C-Cosmos-7B \
       --components-source nvidia/Cosmos-Predict2-2B-Video2World
  3. Provide an input image for 3D-conditioned generation.
"""

import argparse

from fastvideo import VideoGenerator


def main():
    parser = argparse.ArgumentParser(description="GEN3C video generation")
    parser.add_argument("--model_path",
                        type=str,
                        default="converted_weights/GEN3C-Cosmos-7B")
    parser.add_argument("--image_path",
                        type=str,
                        default=None,
                        help="Input image for 3D cache conditioning")
    parser.add_argument("--prompt",
                        type=str,
                        default="A slow camera pan over a sunlit landscape.")
    parser.add_argument(
        "--negative_prompt",
        type=str,
        default=(
            "The video captures a series of frames showing ugly scenes, static with no motion, motion blur, "
            "over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, "
            "underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, "
            "jerky movements, low frame rate, artifacting, color banding, unnatural transitions, outdated special "
            "effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, visual noise, and "
            "flickering. Overall, the video is of poor quality."
        ),
    )
    parser.add_argument("--trajectory",
                        type=str,
                        default="left",
                        choices=[
                            "left", "right", "up", "down", "zoom_in",
                            "zoom_out", "clockwise", "counterclockwise", "none"
                        ])
    parser.add_argument("--movement_distance", type=float, default=0.3)

    args = parser.parse_args()

    # Initialize generator with GEN3C model
    generator = VideoGenerator.from_pretrained(
        args.model_path,
        num_gpus=1, # Adjust as needed
        use_fsdp_inference=True, # Enable FSDP for inference
        dit_cpu_offload=True, # Offload DiT to CPU
        vae_cpu_offload=True, # Offload VAE to CPU
        text_encoder_cpu_offload=True, # Offload text encoder to CPU
        pin_cpu_memory=True, # Pin CPU memory for faster transfers
    )

    # Video generation parameters
    height = 704
    width = 1280
    num_frames = 33

    # Create camera trajectory based on arguments
    # This is a simplified representation; actual trajectory creation might be more complex
    camera_states = create_camera_trajectory(
        action=args.trajectory,
        height=height,
        width=width,
        num_frames=num_frames,
        action_speed=args.movement_distance,
        dtype=torch.bfloat16, # Use bfloat16 for efficiency
    )
    print(f"Camera states shape: {camera_states.shape}")

    # Generate video
    generator.generate_video(
        prompt=args.prompt,
        negative_prompt=args.negative_prompt,
        camera_states=camera_states,
        height=height,
        width=width,
        num_frames=num_frames,
        num_inference_steps=50,
        guidance_scale=6.0,
        seed=42,
        fps=24,
        output_path="video_samples_gen3c",
        save_video=True,
        image_path=args.image_path, # Input image for 3D conditioning
    )


if __name__ == "__main__":
    main()

```

--------------------------------

### Launch FastVideo Server

Source: https://haoailab.com/FastVideo/api/fastvideo/entrypoints/cli/bench_serving?q=

Use this command to start a FastVideo server for multimodal generation models.

```bash
fastvideo serve --config serve.yaml
```

--------------------------------

### Knowledge Distillation YAML Configuration

Source: https://haoailab.com/FastVideo/api/fastvideo/train/methods/knowledge_distillation

Example YAML configuration for the Knowledge Distillation method. This setup defines the student and teacher models, and parameters for the distillation process, including the cache path and timesteps.

```yaml
models:
  student:
    _target_: fastvideo.train.models.wan.WanModel
    init_from: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
    trainable: true
  teacher:           # omit once cache is complete
    _target_: fastvideo.train.models.wan.WanModel
    init_from: Wan-AI/Wan2.1-T2V-14B-Diffusers
    trainable: false
    disable_custom_init_weights: true

method:
  _target_: fastvideo.train.methods.knowledge_distillation.kd.KDMethod
  teacher_path_cache: /data/kd_cache/wan14b_4step
  t_list: [999, 937, 833, 624, 0]   # integer timesteps
  student_sample_steps: 4
  teacher_guidance_scale: 1.0

```

--------------------------------

### Game Initialization and Setup

Source: https://haoailab.com/FastVideo/inference/examples/local?q=

Initializes the game state, including setting up the generator, defining streaming parameters, and preparing action tensors. This function is called when starting a new game session and ensures all necessary components are ready.

```python
        def start_game(model_name, seed_val, randomize, state):
            if randomize:
                seed_val = torch.randint(0, 1000000, (1,)).item()

            config = VARIANT_CONFIG.get(model_name)
            if not config:
                return state, seed_val, "Block: 0 / 50", None, "", gr.update(), gr.update()

            generator = generators.get(config["model_path"])
            if not generator:
                return state, seed_val, "Block: 0 / 50", None, "", gr.update(), gr.update()

            # If already initialized, clean up first
            if state.get("initialized"):
                try:
                    # Clear accumulated frames without saving
                    generator.accumulated_frames = []
                    generator.executor.execute_streaming_clear()
                except Exception as e:
                    print(f"Warning: cleanup error: {e}")

            # Streaming parameters
            num_latent_frames_per_block = 3
            max_blocks = 50
            total_latent_frames = num_latent_frames_per_block * max_blocks
            num_frames = (total_latent_frames - 1) * 4 + 1

            actions = {
                "keyboard": torch.zeros((num_frames, config["keyboard_dim"])),
                "mouse": torch.zeros((num_frames, 2))
            }
            grid_sizes = torch.tensor([150, 44, 80])

            output_dir = os.path.abspath("outputs/matrixgame")
            os.makedirs(output_dir, exist_ok=True)
            video_path = os.path.join(output_dir, f"video_{int(time.time())}.mp4")

            generator.reset(
                prompt="",
                image_path=config["image_url"],
                mouse_cond=actions["mouse"].unsqueeze(0),
                keyboard_cond=actions["keyboard"].unsqueeze(0),
                grid_sizes=grid_sizes,
                num_frames=num_frames,
                height=352,
                width=640,
                num_inference_steps=50,
                output_path=video_path,
            )

            new_state = {
                "initialized": True,
                "current_model": model_name,
                "block_idx": 0,
                "max_blocks": max_blocks,
                "video_path": video_path,
                "frames_per_block": num_latent_frames_per_block * 4,
                "mode": config["mode"],
                "seed": seed_val,
            }

            return new_state, seed_val, "Block: 0 / 50", None, gr.update(value="Step"), gr.update(interactive=True)
```
Tokens:
Raw