ControlFoley

https://github.com/xiaomi-research/controlfoley
Admin
ControlFoley is a unified and controllable video-to-audio generation framework that synthesizes...
Tokens:69,935
Snippets:389
Trust Score:8.4
License:Apache-2.0
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install ControlFoley and download pretrained weights

Source: https://github.com/xiaomi-research/controlfoley/blob/main/README.md

Installation steps for ControlFoley: clone the repo, create a conda environment with Python 3.10.16, install dependencies from requirements.txt, and download the pretrained model weights using the Hugging Face CLI into the 'model_weights' directory.

```bash
# Clone the repository
git clone https://github.com/xiaomi-research/controlfoley
cd controlfoley

# Create conda environment
conda create -n controlfoley python=3.10.16
conda activate controlfoley

# Install dependencies
pip install -r requirements.txt

# Download pretrained weights
pip install huggingface-hub==0.26.2
huggingface-cli download YJX-Xiaomi/ControlFoley --resume-download --local-dir model_weights --local-dir-use-symlinks False
```

--------------------------------

### Run V2A inference with video only

Source: https://github.com/xiaomi-research/controlfoley/blob/main/README.md

V2A (Video-to-Audio) example: only a video file is required. Generates audio corresponding to the video content without a text prompt.

```bash
python demo.py --video "examples/v2a/v2a_video_input.mp4" --duration 6.0 --output "./output"
```

--------------------------------

### Run T2A inference with prompt only

Source: https://github.com/xiaomi-research/controlfoley/blob/main/README.md

T2A (Text-to-Audio) example: only a textual prompt is used to generate audio. No video or reference audio is needed.

```bash
python demo.py --prompt "A bird sings melodically in a forest." --duration 10.0 --output "./output"
```

--------------------------------

### Run TV2A inference with video and prompt

Source: https://github.com/xiaomi-research/controlfoley/blob/main/README.md

TV2A (Text-Video-to-Audio) example: requires a video file and a prompt, with optional duration. Generates audio matching the video content and the text prompt.

```bash
python demo.py --video "examples/tv2a/tv2a_video_input.mp4" --prompt "skateboarding" --duration 8.0 --output "./output"
```

--------------------------------

### Run AC-V2A inference with video and reference audio

Source: https://github.com/xiaomi-research/controlfoley/blob/main/README.md

AC-V2A (Audio-Conditioned Video-to-Audio) example: requires both video and reference audio inputs. The audio content is used to condition the generation.

```bash
python demo.py --video "examples/ac_v2a/acv2a_video_input.mp4" --audio "examples/ac_v2a/acv2a_reference.wav" --duration 5.0 --output "./output"
```

--------------------------------

### Run TC-V2A inference with video and prompt

Source: https://github.com/xiaomi-research/controlfoley/blob/main/README.md

TC-V2A (Text-Conditioned Video-to-Audio) example: uses a video file and a textual prompt to generate audio. The prompt describes the desired sound effect.

```bash
python demo.py --video "examples/tc_v2a/tcv2a_video_input.mp4" --prompt "thunder strike" --duration 6.0 --output "./output"
```

--------------------------------

### demo.py CLI options

Source: https://github.com/xiaomi-research/controlfoley/blob/main/README.md

Command-line usage for demo.py: lists all options with their types and defaults. Use this to run inference with video, audio, prompt, negative prompt, duration, and output directory.

```text
python demo.py [OPTIONS]

Options:
  --video            TEXT       Path to the input video file. (default: None)
  --audio            TEXT       Path to the input reference audio file. (default: None)
  --prompt           TEXT       Textual prompt for audio generation. (default: None)
  --negative_prompt  TEXT       Negative textual prompt for audio generation. (default: None)
  --duration         FLOAT      Duration of the generated audio in seconds. (default: 8.0)
  --output           TEXT       Output directory for generated audio files. (default: ./output)
```
Tokens:
Raw