Stable Virtual Camera

https://github.com/stability-ai/stable-virtual-camera
Admin
Stable Virtual Camera (SEVA) is a generative diffusion model for Novel View Synthesis, capable of...
Tokens:8,000
Snippets:29
Trust Score:9.3
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Development Dependencies

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/INSTALL.md

Installs optional dependencies for development, including pre-commit hooks.

```bash
# [OPTIONAL] Install seva dependencies for development.
pip install -e ".[dev]"
pre-commit install
```

--------------------------------

### Install Demo Dependencies

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/INSTALL.md

Installs dependencies needed for the command-line and Gradio demos, including pycolmap and dust3r.

```bash
# Initialize and update submodules for demo.
git submodule update --init --recursive

# Install pycolmap dependencies for cli and gradio demo (our model is not dependent on it).
echo "Installing pycolmap (for both cli and gradio demo)..."
pip install git+https://github.com/jensenz-sai/pycolmap@543266bc316df2fe407b3a33d454b310b1641042

# Install dust3r dependencies for gradio demo (our model is not dependent on it).
echo "Installing dust3r dependencies (only for gradio demo)..."
pushd third_party/dust3r
pip install -r requirements.txt
popd
```

--------------------------------

### Install Model Dependencies

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/INSTALL.md

Installs the core dependencies required for the Stable Virtual Camera model using pip.

```bash
# Install seva model dependencies.
pip install -e .
```

--------------------------------

### Example img2img run on dl3d140-165f5af8bfe32f70595a1c9393a6e442acf7af019998275144f605b89a306557

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This is a concrete example of running the img2img command on a specific dataset instance ('dl3d140-165f5af8bfe32f70595a1c9393a6e442acf7af019998275144f605b89a306557') with 3 input views, demonstrating practical application of the parameters.

```bash
python demo.py \
    --data_path /path/to/assets_demo_cli/ \
    --data_items dl3d140-165f5af8bfe32f70595a1c9393a6e442acf7af019998275144f605b89a306557 \
    --num_inputs 3 \
    --video_save_fps 10
```

--------------------------------

### Install PyTorch Nightly for Speed-up

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/INSTALL.md

Installs the PyTorch nightly version to leverage torch.compile for faster JIT compilation, potentially doubling sampling speed. Users should adjust the CUDA version in the command to match their system.

```bash
# [OPTIONAL] Install the torch nightly version for faster JIT via. torch.compile (speed up sampling by 2x in our testing).
# Please adjust to your own cuda version. For example, if you have cuda 11.8, use the following command.
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118
```

--------------------------------

### Run CLI Demo

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

Executes the CLI demo script with specified data path and additional arguments. This is the primary command to start the demo.

```bash
python demo.py --data_path <data_path> [additional arguments]
```

--------------------------------

### Install Stable Virtual Camera

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/README.md

Clones the repository and installs the Stable Virtual Camera package. Requires Python 3.10+ and PyTorch 2.6.0+.

```bash
git clone --recursive https://github.com/Stability-AI/stable-virtual-camera
cd stable-virtual-camera
pip install -e .
```

--------------------------------

### Run img2img for single-view conditioning with camera scale sweep

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This command illustrates the setup for single-view conditioning tests, where a range of camera scales are applied to find optimal parameters. It also shows how to adjust the 'cfg' parameter for datasets like RealEstate10K to improve results.

```bash
python demo.py \
    --data_path <data_path> \
    --num_inputs <P> \
    --video_save_fps 10 \
    --camera_scale <camera_scale> \
    --cfg 6.0
```

--------------------------------

### Data Conversion Script Example

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/benchmark/README.md

Provides an example script for converting existing academic datasets into the 'reconfusion' format, which is compatible with ReconfusionParser. This script is useful for preparing data for the Stable Virtual Camera project.

```Python
import os

def convert_to_reconfusion(dataset_path, output_path):
    # Placeholder for conversion logic
    print(f"Converting dataset from {dataset_path} to {output_path}")
    # Example: Copying files and creating necessary metadata
    pass

# Example usage:
# convert_to_reconfusion('path/to/academic/dataset', 'path/to/reconfusion/format')

```

--------------------------------

### Run img2img for semi-dense-view regime with extended T

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This example configures the img2img process for semi-dense-view scenarios on datasets like DL3DV-140 and Tanks and Temples. It involves extending the 'T' parameter to accommodate all input and target views in a single forward pass.

```bash
python demo.py \
    --data_path <data_path> \
    --num_inputs 32 \
    --video_save_fps 10 \
    --T 90
```

--------------------------------

### Run img2img with trajectory prior and nearest-gt chunking

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This example shows how to leverage trajectory priors for improved 3D consistency, specifically using the 'orbit' prior for CO3D dataset with the 'nearest-gt' chunking strategy. This is beneficial when trajectory information is available.

```bash
python demo.py \
    --data_path <data_path> \
    --num_inputs <P> \
    --video_save_fps 10 \
    --use_traj_prior True \
    --traj_prior orbit \
    --chunking_strategy nearest-gt
```

--------------------------------

### Run CLI Demo

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/README.md

Execute the command-line interface demo for the Stable Virtual Camera. This command allows for passing data paths and additional arguments for fine-grained control over the model, suitable for advanced users and researchers.

```bash
python demo.py --data_path <data_path> [additional arguments]
```

--------------------------------

### Run Gradio Demo

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/README.md

Launches the Gradio demo for Stable Virtual Camera, providing a GUI interface for users.

```python
python demo_gr.py
```

--------------------------------

### Run Gradio Demo Locally

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/GR_USAGE.md

This command initiates the Gradio demo application locally, allowing users to interact with the Stable Virtual Camera project.

```bash
python demo_gr.py
```

--------------------------------

### Run img2img with specified data path and number of inputs

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This command demonstrates the basic usage of the img2img functionality. It requires a data path and optionally the number of inputs if multiple train_test_split_*.json files are present. Additional parameters can be set for trajectory priors, chunking strategies, and camera scales depending on the dataset and desired output quality.

```bash
python demo.py \
    --data_path <data_path> \
    --num_inputs <P> \
    --video_save_fps 10
```

--------------------------------

### Hugging Face Authentication

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/README.md

Logs into Hugging Face CLI to authenticate and download model weights. This is necessary for the Stable Virtual Camera to function.

```bash
# This will prompt you to enter your Hugging Face credentials.
huggingface-cli login
```

--------------------------------

### Run img2img for ViewCrafter split with extended T and L_short

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This command is tailored for the ViewCrafter split, optimizing for original image resolutions by extending 'T' and setting 'L_short'. This approach is beneficial for datasets like RealEastate10K, CO3D, and Tanks and Temples.

```bash
python demo.py \
    --data_path <data_path> \
    --num_inputs <P> \
    --video_save_fps 10 \
    --T 25 \
    --L_short 576
```

--------------------------------

### Image Preprocessing and Postprocessing

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/benchmark/README.md

Details the image preprocessing steps applied to model inputs and postprocessing steps applied to model outputs for evaluation and benchmarking. These operations ensure data consistency and prepare samples for metric computation.

```General
Image Preprocessing: resize the shortest side to 576 (--L_short 576), center crop
Image Postprocessing: center crop to 576
```

--------------------------------

### Benchmark Dataset Details

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/benchmark/README.md

This snippet outlines the structure and content of the benchmark dataset used for evaluating NVS models. It details various datasets, their splits, file paths, content descriptions, and image processing steps.

```Markdown
# :bar_chart: Benchmark

We provide <a href="https://github.com/Stability-AI/stable-virtual-camera/releases/tag/benchmark">in this release</a> (`benchmark.zip`) with the following 17 entries as a benchmark to evaluate NVS models.
We hope this will help standardize the evaluation of NVS models and facilitate fair comparison between different methods.

<table>
  <thead>
    <tr>
      <th align="center">Dataset</th>
      <th align="center">Split</th>
      <th align="center">Path</th>
      <th align="center">Content</th>
      <th align="center">Image Preprocessing</th>
      <th align="center">Image Postprocessing</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td align="center">OmniObject3D</td>
      <td align="center"><code>S</code> (SV3D), <code>O</code> (Ours) </td>
      <td align="center"><code>omniobject3d</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center">GSO</td>
      <td align="center"><code>S</code> (SV3D), <code>O</code> (Ours) </td>
      <td align="center"><code>gso</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center" rowspan="4">RealEstate10K</td>
      <td align="center"><code>D</code> (4DiM) </td>
      <td align="center"><code>re10k-4dim</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">resize to 256</td>
    </tr>
    <tr>
      <td align="center"><code>R</code> (ReconFusion) </td>
      <td align="center"><code>re10k</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center"><code>P</code> (pixelSplat) </td>
      <td align="center"><code>re10k-pixelsplat</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">resize to 256</td>
    </tr>
    <tr>
      <td align="center"><code>V</code> (ViewCrafter) </td>
      <td align="center"><code>re10k-viewcrafter</code></td>
      <td align="center"><code>images/*.png</code>,<code>transforms.json</code>,<code>train_test_split_*.json</code></td>
      <td align="center">resize the shortest side to 576 (<code>--L_short 576</code>)</td>
      <td align="center">center crop</td>
    </tr>
    <tr>
      <td align="center">LLFF</td>
      <td align="center"><code>R</code> (ReconFusion) </td>
      <td align="center"><code>llff</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center">DTU</td>
      <td align="center"><code>R</code> (ReconFusion) </td>
      <td align="center"><code>dtu</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center" rowspan="2">CO3D</td>
      <td align="center"><code>R</code> (ReconFusion) </td>
      <td align="center"><code>co3d</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center"><code>V</code> (ViewCrafter) </td>
      <td align="center"><code>co3d-viewcrafter</code></td>
      <td align="center"><code>images/*.png</code>,<code>transforms.json</code>,<code>train_test_split_*.json</code></td>
      <td align="center">resize the shortest side to 576 (<code>--L_short 576</code>)</td>
      <td align="center">center crop</td>
    </tr>
    <tr>
      <td align="center" rowspan="2" >WildRGB-D</td>
      <td align="center"><code>Oₑ</code> (Ours, easy) </td>
      <td align="center"><code>wildgbd/easy</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center"><code>Oₕ</code> (Ours, hard) </td>
      <td align="center"><code>wildgbd/hard</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center">Mip-NeRF360</td>
      <td align="center"><code>R</code> (ReconFusion) </td>
      <td align="center"><code>mipnerf360</code></td>
      <td align="center"><code>train_test_split_*.json</code></td>
      <td align="center">center crop to 576</td>
      <td align="center">\</td>
    </tr>
    <tr>
      <td align="center" rowspan="2">DL3DV-140</td>

```

--------------------------------

### Run img2vid for ViewCrafter split with extended T and L_short

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

Similar to img2img, this command configures img2vid for the ViewCrafter split by extending 'T' and setting 'L_short' to match original image resolutions, ensuring better performance on datasets like RealEastate10K, CO3D, and Tanks and Temples.

```bash
python demo.py \
    --data_path <data_path> \
    --task img2vid \
    --replace_or_include_input True \
    --num_inputs <P> \
    --use_traj_prior True \
    --chunk_strategy interp \
    --T 25 \
    --L_short 576
```

--------------------------------

### Cite Stable Virtual Camera

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/README.md

BibTeX entry for citing the Stable Virtual Camera project, which utilizes diffusion models for generative view synthesis. This citation is useful for academic and research purposes.

```bibtex
@article{zhou2025stable,
    title={Stable Virtual Camera: Generative View Synthesis with Diffusion Models},
    author={Jensen (Jinghao) Zhou and Hang Gao and Vikram Voleti and Aaryaman Vasishta and Chun-Han Yao and Mark Boss and
    Philip Torr and Christian Rupprecht and Varun Jampani
    },
    journal={arXiv preprint arXiv:2503.14489},
    year={2025}
}
```

--------------------------------

### Run img2trajvid in sparse-view regime with specific cfg and chunking

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This command demonstrates the img2trajvid task in a sparse-view regime (P<=8). It specifies 'cfg' values for multiple passes and uses 'interp-gt' as the chunking strategy, with an option for 'gt-nearest' for larger numbers of input frames.

```bash
python demo.py \
    --data_path <data_path> \
    --task img2trajvid \
    --num_inputs <P> \
    --cfg 3.0,2.0  \
    --use_traj_prior True \
    --chunk_strategy interp-gt
```

--------------------------------

### Data Format for img2trajvid_s-prob

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

Specifies the directory structure for the 'img2trajvid_s-prob' task, which requires a single input image.

```bash
<data_path>/
  ├── scene_1.png
  ├── scene_2.png
  └── scene_3.png
```

--------------------------------

### Generate Trajectory Video from Images (Bash)

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This bash command executes the `demo.py` script to generate a trajectory video from input images. It requires specifying the data path, task as 'img2trajvid', the number of input views, and configuration parameters like chunking strategy and sequence length.

```bash
python demo.py \
    --data_path <data_path> \
    --task img2trajvid \
    --num_inputs <P> \
    --cfg 3.0  \
    --L_short 576 \
    --use_traj_prior True \
    --chunk_strategy interp
```

--------------------------------

### Run img2vid with input view replacement and interpolation chunking

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This command utilizes the img2vid task, which requires replacing or including input views to form a complete trajectory. It uses the 'interp' chunking strategy by default and can be configured with trajectory priors and extended 'T' values for specific datasets.

```bash
python demo.py \
    --data_path <data_path> \
    --task img2vid \
    --replace_or_include_input True \
    --num_inputs <P> \
    --use_traj_prior True \
    --chunk_strategy interp
```

--------------------------------

### Data Format for Other Tasks

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

Defines the directory structure for tasks other than 'img2trajvid_s-prob'. It includes scene subdirectories with images, transforms.json, and train_test_split files.

```bash
<data_path>/
├── scene_1/
    ├── train_test_split_1.json # for single-view regime
    ├── train_test_split_6.json # for sparse-veiw regime
    ├── train_test_split_32.json # for semi-dense-view regime
    ├── transforms.json
    └── images/
        ├── image_0.png
        ├── image_1.png
        ├── ...
        └── image_1000.png
├── scene_2
└── scene_3
```

--------------------------------

### Run img2trajvid_s-prob with orbit prior and adjusted parameters

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

This command showcases the img2trajvid_s-prob task, utilizing an 'orbit' trajectory prior. It includes adjustments for 'cfg', 'guider', 'num_targets', and 'L_short', along with options to tune 'camera_scale' for panning motions.

```bash
python demo.py \
    --data_path <data_path> \
    --task img2trajvid_s-prob \
    --replace_or_include_input True \
    --traj_prior orbit \
    --cfg 4.0,2.0 \
    --guider 1,2 \
    --num_targets 111 \
    --L_short 576 \
    --use_traj_prior True \
    --chunk_strategy interp
```

--------------------------------

### Configure Procedural Sampling

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

Enables procedural sampling by setting specific flags. This is recommended for controlling the sampling process.

```bash
python demo.py --use_traj_prior True --chunk_strategy <chunk_strategy>
```

--------------------------------

### Set Custom Resolution

Source: https://github.com/stability-ai/stable-virtual-camera/blob/main/docs/CLI_USAGE.md

Overrides default image preprocessing to set a custom width and height for input and output images. Alternatively, setting `--L_short` maintains the aspect ratio.

```bash
python demo.py --W <W> --H <H>
```

```bash
python demo.py --L_short 576
```
Tokens:
Raw