### Environment Setup for AudioX Source: https://github.com/zeyuet/audiox/blob/main/README.md Commands to clone the AudioX repository, set up a Conda environment, install necessary Python packages including the AudioX library, and install system dependencies like ffmpeg and libsndfile. ```bash git clone https://github.com/ZeyueT/AudioX.git cd AudioX conda create -n AudioX python=3.8.20 conda activate AudioX pip install git+https://github.com/ZeyueT/AudioX.git conda install -c conda-forge ffmpeg libsndfile ``` -------------------------------- ### AudioX Script Inference Example Source: https://github.com/zeyuet/audiox/blob/main/README.md Python code demonstrating how to set up the device (GPU or CPU) and import necessary libraries for inference with the AudioX model, including PyTorch, torchaudio, einops, and components from stable_audio_tools. ```python import torch import torchaudio from einops import rearrange from stable_audio_tools import get_pretrained_model from stable_audio_tools.inference.generation import generate_diffusion_cond from stable_audio_tools.data.utils import read_video, merge_video_audio from stable_audio_tools.data.utils import load_and_process_audio import os device = "cuda" if torch.cuda.is_available() else "cpu" ``` -------------------------------- ### Launch Gradio Demo for AudioX Source: https://github.com/zeyuet/audiox/blob/main/README.md Python command to launch the Gradio interactive demo for the AudioX model locally. It requires the model configuration file and can optionally share the demo publicly. ```bash python3 run_gradio.py \ --model-config model/config.json \ --share ``` -------------------------------- ### Download Pretrained AudioX Model Source: https://github.com/zeyuet/audiox/blob/main/README.md Shell commands to download the pretrained AudioX model checkpoint and configuration file from Hugging Face. ```bash mkdir -p model wget https://huggingface.co/HKUSTAudio/AudioX/resolve/main/model.ckpt -O model/model.ckpt wget https://huggingface.co/HKUSTAudio/AudioX/resolve/main/config.json -O model/config.json ``` -------------------------------- ### Video-to-Music Generation with AudioX Model Source: https://github.com/zeyuet/audiox/blob/main/README.md This snippet demonstrates the end-to-end process of generating music from a video using the AudioX pretrained model. It includes loading the model, preparing video and audio tensors, conditioning the generation process with video and text prompts, generating stereo audio, and saving the output as a WAV file. Finally, it merges the generated audio with the original video. ```python from audiox import get_pretrained_model, read_video, load_and_process_audio, generate_diffusion_cond from einops.rearrange import rearrange import torch import torchaudio import os # Assume 'device' is defined, e.g., device = torch.device("cuda" if torch.cuda.is_available() else "cpu") device = "cpu" # Example device # Download model model, model_config = get_pretrained_model("HKUSTAudio/AudioX") sample_rate = model_config["sample_rate"] sample_size = model_config["sample_size"] target_fps = model_config["video_fps"] seconds_start = 0 seconds_total = 10 model = model.to(device) # for video-to-music generation video_path = "example/V2M_sample-1.mp4" text_prompt = "Generate music for the video" audio_path = None video_tensor = read_video(video_path, seek_time=0, duration=seconds_total, target_fps=target_fps) audio_tensor = load_and_process_audio(audio_path, sample_rate, seconds_start, seconds_total) conditioning = [{ "video_prompt": [video_tensor.unsqueeze(0)], "text_prompt": text_prompt, "audio_prompt": audio_tensor.unsqueeze(0), "seconds_start": seconds_start, "seconds_total": seconds_total }] # Generate stereo audio output = generate_diffusion_cond( model, steps=250, cfg_scale=7, conditioning=conditioning, sample_size=sample_size, sigma_min=0.3, sigma_max=500, sampler_type="dpmpp-3m-sde", device=device ) # Rearrange audio batch to a single sequence output = rearrange(output, "b d n -> d (b n)") # Peak normalize, clip, convert to int16, and save to file output = output.to(torch.float32).div(torch.max(torch.abs(output))).clamp(-1, 1).mul(32767).to(torch.int16).cpu() torchaudio.save("output.wav", output, sample_rate) # Merge video and audio if video exists if video_path is not None and os.path.exists(video_path): # Assume merge_video_audio is a function available in the environment # merge_video_audio(video_path, "output.wav", "output.mp4", 0, seconds_total) print("Video and audio merging would happen here if merge_video_audio was defined.") else: print("Video path not found or not specified, skipping video-audio merge.") ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.