AudioX

https://github.com/zeyuet/audiox
Admin
AudioX is a unified Diffusion Transformer model for anything-to-audio and music generation, offering...
Tokens:1,062
Snippets:5
Trust Score:6.7
Update:1 year ago
Show doc for...
Tokens:
Raw
### Environment Setup for AudioX

Source: https://github.com/zeyuet/audiox/blob/main/README.md

Commands to clone the AudioX repository, set up a Conda environment, install necessary Python packages including the AudioX library, and install system dependencies like ffmpeg and libsndfile.

```bash
git clone https://github.com/ZeyueT/AudioX.git
cd AudioX
conda create -n AudioX python=3.8.20
conda activate AudioX
pip install git+https://github.com/ZeyueT/AudioX.git
conda install -c conda-forge ffmpeg libsndfile
```

--------------------------------

### AudioX Script Inference Example

Source: https://github.com/zeyuet/audiox/blob/main/README.md

Python code demonstrating how to set up the device (GPU or CPU) and import necessary libraries for inference with the AudioX model, including PyTorch, torchaudio, einops, and components from stable_audio_tools.

```python
import torch
import torchaudio
from einops import rearrange
from stable_audio_tools import get_pretrained_model
from stable_audio_tools.inference.generation import generate_diffusion_cond
from stable_audio_tools.data.utils import read_video, merge_video_audio
from stable_audio_tools.data.utils import load_and_process_audio
import os

device = "cuda" if torch.cuda.is_available() else "cpu"

```

--------------------------------

### Launch Gradio Demo for AudioX

Source: https://github.com/zeyuet/audiox/blob/main/README.md

Python command to launch the Gradio interactive demo for the AudioX model locally. It requires the model configuration file and can optionally share the demo publicly.

```bash
python3 run_gradio.py \
    --model-config model/config.json \
    --share
```

--------------------------------

### Download Pretrained AudioX Model

Source: https://github.com/zeyuet/audiox/blob/main/README.md

Shell commands to download the pretrained AudioX model checkpoint and configuration file from Hugging Face.

```bash
mkdir -p model
wget https://huggingface.co/HKUSTAudio/AudioX/resolve/main/model.ckpt -O model/model.ckpt
wget https://huggingface.co/HKUSTAudio/AudioX/resolve/main/config.json -O model/config.json
```

--------------------------------

### Video-to-Music Generation with AudioX Model

Source: https://github.com/zeyuet/audiox/blob/main/README.md

This snippet demonstrates the end-to-end process of generating music from a video using the AudioX pretrained model. It includes loading the model, preparing video and audio tensors, conditioning the generation process with video and text prompts, generating stereo audio, and saving the output as a WAV file. Finally, it merges the generated audio with the original video.

```python
from audiox import get_pretrained_model, read_video, load_and_process_audio, generate_diffusion_cond
from einops.rearrange import rearrange
import torch
import torchaudio
import os

# Assume 'device' is defined, e.g., device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
device = "cpu" # Example device

# Download model
model, model_config = get_pretrained_model("HKUSTAudio/AudioX")
sample_rate = model_config["sample_rate"]
sample_size = model_config["sample_size"]
target_fps = model_config["video_fps"]
seconds_start = 0
seconds_total = 10

model = model.to(device)

# for video-to-music generation
video_path = "example/V2M_sample-1.mp4"
text_prompt = "Generate music for the video" 
audio_path = None

video_tensor = read_video(video_path, seek_time=0, duration=seconds_total, target_fps=target_fps)
audio_tensor = load_and_process_audio(audio_path, sample_rate, seconds_start, seconds_total)

conditioning = [{
    "video_prompt": [video_tensor.unsqueeze(0)],        
    "text_prompt": text_prompt,
    "audio_prompt": audio_tensor.unsqueeze(0),
    "seconds_start": seconds_start,
    "seconds_total": seconds_total
}]
    
# Generate stereo audio
output = generate_diffusion_cond(
    model,
    steps=250,
    cfg_scale=7,
    conditioning=conditioning,
    sample_size=sample_size,
    sigma_min=0.3,
    sigma_max=500,
    sampler_type="dpmpp-3m-sde",
    device=device
)

# Rearrange audio batch to a single sequence
output = rearrange(output, "b d n -> d (b n)")

# Peak normalize, clip, convert to int16, and save to file
output = output.to(torch.float32).div(torch.max(torch.abs(output))).clamp(-1, 1).mul(32767).to(torch.int16).cpu()
torchaudio.save("output.wav", output, sample_rate)

# Merge video and audio if video exists
if video_path is not None and os.path.exists(video_path):
    # Assume merge_video_audio is a function available in the environment
    # merge_video_audio(video_path, "output.wav", "output.mp4", 0, seconds_total)
    print("Video and audio merging would happen here if merge_video_audio was defined.")
else:
    print("Video path not found or not specified, skipping video-audio merge.")

```
Tokens:
Raw