### Launch Gradio Demo

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

Run this command to start the Gradio interface for FoleyCrafter. The --share flag makes the demo publicly accessible.

```bash
python app.py --share
```

--------------------------------

### Install FoleyCrafter Dependencies

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

Use this command to create and activate the conda environment for FoleyCrafter. Ensure you have conda installed.

```bash
conda env create -f requirements/environment.yaml
conda activate foleycrafter

conda install git-lfs
git lfs install
```

--------------------------------

### Full Batch CLI and Programmatic Inference

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Demonstrates how to run the complete FoleyCrafter pipeline for batch processing of video files. This includes setting up arguments via CLI or programmatically, building the necessary models, and executing the inference loop. Outputs generated WAV files and muxes audio back into original videos.

```python
# CLI usage
# python inference.py \
#   --input=examples/sora/ \
#   --save_dir=output/sora/ \
#   --seed=42 \
#   --prompt="" \
#   --nprompt="" \
#   --semantic_scale=1.0 \
#   --temporal_scale=0.2

# Equivalent programmatic call:
import argparse
from inference import args_parse, build_models, run_inference

config = argparse.Namespace(
    prompt="",
    nprompt="",
    seed=42,
    semantic_scale=1.0,
    temporal_scale=0.2,
    input="examples/sora",
    ckpt="checkpoints/",
    save_dir="output/sora/",
    pretrain="auffusion/auffusion-full-no-adapter",
    device="cuda",
)

pipe, vocoder, time_detector = build_models(config)
run_inference(config, pipe, vocoder, time_detector)
# Outputs:
#   output/sora/audio/0.wav  (16kHz mono WAV)
#   output/sora/video/0.mp4  (original video with generated audio)
```

--------------------------------

### Temporal Alignment with Visual Cues

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

This command enables temporal alignment with visual cues for audio generation. Ensure the input directory contains the necessary files for alignment.

```bash
python inference.py \
--temporal_align \
--input=input/avsync \
--save_dir=output/avsync/
```

--------------------------------

### Load FoleyCrafter Model Components

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Downloads and loads the Auffusion base model, FoleyCrafter checkpoint, HiFi-GAN vocoder, VideoOnsetNet, temporal ControlNet adapter, and semantic IP-Adapter weights. Returns the pipeline, vocoder, and time detector needed for inference. Ensure the 'checkpoints/' directory exists for saving downloaded weights.

```python
import argparse
from inference import build_models

config = argparse.Namespace(
    pretrain="auffusion/auffusion-full-no-adapter",
    ckpt="checkpoints/",
    device="cuda",
    semantic_scale=1.0,
    temporal_scale=0.2,
)

pipe, vocoder, time_detector = build_models(config)
# pipe      -> StableDiffusionControlNetPipeline on CUDA
# vocoder   -> Generator (HiFi-GAN) on CUDA
# time_detector -> VideoOnsetNet on CUDA
```

--------------------------------

### Initialize VideoOnsetNet for Temporal Event Detection

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Loads the VideoOnsetNet, a video onset detection network based on R2+1D-18, for predicting per-frame onset probabilities from video frames. Ensure the 'checkpoints/timestamp_detector.pth.tar' file is available. The network requires normalized video frames as input and outputs probabilities that are thresholded at 0.5 to detect events.

```python
import torch
import torchvision
from foleycrafter.models.time_detector.model import VideoOnsetNet
from foleycrafter.models.onset import torch_utils

# Load the timestamp detector
time_detector = VideoOnsetNet(pretrained=False)
time_detector, _ = torch_utils.load_model(
    "checkpoints/timestamp_detector.pth.tar",
    time_detector,
    device="cuda",
    strict=True,
)
time_detector.eval()

# Prepare video frames tensor (150 frames, already normalized)
vision_transform = torchvision.transforms.Compose([
    torchvision.transforms.Resize((128, 128)),
    torchvision.transforms.CenterCrop((112, 112)),
    torchvision.transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    ),
])
```

--------------------------------

### Inference Script Options

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

This section lists the available command-line options for the inference script, including prompts, seeds, and directory configurations.

```console
options:
  -h, --help            show this help message and exit
  --prompt PROMPT       prompt for audio generation
  --nprompt NPROMPT     negative prompt for audio generation
  --seed SEED           ramdom seed
  --temporal_align TEMPORAL_ALIGN
                        use temporal adapter or not
  --temporal_scale TEMPORAL_SCALE
                        temporal align scale
  --semantic_scale SEMANTIC_SCALE
                        visual content scale
  --input INPUT         input video folder path
  --ckpt CKPT           checkpoints folder path
  --save_dir SAVE_DIR   generation result save path
  --pretrain PRETRAIN   generator checkpoint path
  --device DEVICE
```

--------------------------------

### Generate Audio from Video

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

Use this command to generate audio from a video file. Specify the output directory for the generated audio.

```bash
python inference.py --save_dir=output/sora/
```

--------------------------------

### Generate Audio with Positive Prompt

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

Use this command to generate audio based on a text prompt. Ensure the input directory and save directory are correctly specified. The `prompt` argument controls the audio content.

```bash
python inference.py \
--input=input/PromptControl/case1/ \
--seed=10201304011203481429 \
--save_dir=output/PromptControl/case1/

python inference.py \
--input=input/PromptControl/case1/ \
--seed=10201304011203481429 \
--prompt='noisy, people talking' \
--save_dir=output/PromptControl/case1_prompt/

python inference.py \
--input=input/PromptControl/case2/ \
--seed=10021049243103289113 \
--save_dir=output/PromptControl/case2/

python inference.py \
--input=input/PromptControl/case2/ \
--seed=10021049243103289113 \
--prompt='seagulls' \
--save_dir=output/PromptControl/case2_prompt/
```

--------------------------------

### Download Auffusion Base Model

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

Clone the Auffusion text-to-audio base model from Hugging Face. This is required for FoleyCrafter's functionality.

```bash
git clone https://huggingface.co/auffusion/auffusion-full-no-adapter checkpoints/auffusion
```

--------------------------------

### Gradio Web Interface Controller

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Shows how to use the `FoleyController` class to manage model loading and inference for a Gradio web application. The `foley()` method accepts video file paths and various UI parameters to generate an output video with synthesized audio. Launch the demo using `python app.py --share`.

```python
# Launch the Gradio demo (serves on http://0.0.0.0:7860 by default)
# python app.py --share

# The FoleyController.foley() method signature mirrors the UI inputs:
result_video_path = controller.foley(
    input_video="examples/gen3/case1.mp4",
    prompt_textbox="",                 # optional text prompt
    negative_prompt_textbox="",        # optional negative prompt
    ip_adapter_scale=1.0,             # visual content scale (0–1)
    temporal_scale=0.2,               # temporal alignment scale (0–1)
    sampler_dropdown="DDIM",          # DDIM | Euler | PNDM
    sample_step_slider=25,            # denoising steps (10–100)
    cfg_scale_slider=7.5,             # classifier-free guidance scale
    seed_textbox="33817921",          # reproducibility seed
)
print(result_video_path)
# "samples/Gradio-2024-07-01T00-00-00/sample/output.mp4"
```

--------------------------------

### Run Inference Script

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

Use this command to run the inference script for audio generation. Specify input, seed, and save directory.

```bash
python inference.py \
--input=input/PromptControl/case4/ \
--seed=10014024412012338096 \
--save_dir=output/PromptControl/case4/
```

```bash
python inference.py \
--input=input/PromptControl/case4/ \
--seed=10014024412012338096 \
--nprompt='noisy, wind noise' \
--save_dir=output/PromptControl/case4_nprompt/
```

--------------------------------

### Download FoleyCrafter Checkpoints

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

Clone the FoleyCrafter model from Hugging Face. Place the checkpoints in the specified directory structure.

```bash
git clone https://huggingface.co/ymzhang319/FoleyCrafter checkpoints/
```

--------------------------------

### Generate Audio with Negative Prompt

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

This command demonstrates generating audio while excluding specific content using a negative prompt. The `nprompt` argument specifies the content to avoid.

```bash
python inference.py \
--input=input/PromptControl/case3/ \
--seed=10041042941301238011 \
--save_dir=output/PromptControl/case3/

python inference.py \
--input=input/PromptControl/case3/ \
--seed=10041042941301238011 \
--nprompt='river flows' \
--save_dir=output/PromptControl/case3_nprompt/
```

--------------------------------

### Run StableDiffusionControlNetPipeline for Audio Generation

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Executes the full audio generation pipeline conditioned on text, CLIP image embeddings, and a temporal onset map. Produces a spectrogram image tensor.

```python
import torch

generator = torch.Generator(device="cuda")
generator.manual_seed(42)

sample = pipe(
    prompt="water splashing, waves",          # optional text guidance
    negative_prompt="music, speech",          # optional negative guidance
    ip_adapter_image_embeds=image_embeddings, # CLIP semantic conditioning
    image=time_condition,                     # ControlNet temporal conditioning
    controlnet_conditioning_scale=0.2,        # temporal alignment strength
    num_inference_steps=25,
    height=256,
    width=1024,
    output_type="pt",
    generator=generator,
)

audio_img = sample.images[0]   # torch.Tensor (3, 256, 1024)
print(audio_img.shape)         # torch.Size([3, 256, 1024])
print(audio_img.min(), audio_img.max())
```

--------------------------------

### Process Video Frames for Temporal Detection

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Prepares video frames for temporal onset detection. Ensures frames are in the correct PyTorch tensor format and applies a vision transform.

```python
time_frames = torch.FloatTensor(frames).permute(0, 3, 1, 2)  # (150, 3, H, W)
time_frames = vision_transform(time_frames)
time_frames = {"frames": time_frames.unsqueeze(0).permute(0, 2, 1, 3, 4)}  # (1, 3, 150, 112, 112)

with torch.no_grad():
    preds = time_detector(time_frames)    # (1, 150)
    preds = torch.sigmoid(preds)

print(preds.shape)   # torch.Size([1, 150])
print((preds > 0.5).sum().item())  # number of detected onset frames
```

--------------------------------

### Build ControlNet Input from Onset Predictions

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Converts onset probabilities into a temporal conditioning map for ControlNet. Bins with onset probability > 0.5 are set to +1.0, others to -1.0. The output tensor has a shape of (1, 1, 256, 1024).

```python
import torch

duration = min(duration, 10.0)  # cap at 10 seconds

time_condition = [
    -1 if preds[0][int(i / (1024 / 10 * duration) * 150)] < 0.5 else 1
    for i in range(int(1024 / 10 * duration))
]
time_condition = time_condition + [-1] * (1024 - len(time_condition))

# Shape: (1, 1, 256, 1024) — broadcast over height dimension
time_condition = (
    torch.FloatTensor(time_condition)
    .unsqueeze(0).unsqueeze(0).unsqueeze(0)
    .repeat(1, 1, 256, 1)
    .to("cuda")
)
print(time_condition.shape)   # torch.Size([1, 1, 256, 1024])
```

--------------------------------

### Build FoleyCrafter Diffusion Pipeline

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Constructs the full StableDiffusionControlNetPipeline for FoleyCrafter from a pretrained Auffusion checkpoint. Returns the pipeline ready for adapter loading. Ensure the base model is downloaded or cached locally.

```python
from foleycrafter.utils.util import build_foleycrafter

# Build the pipeline from the default Auffusion base model
# (downloads from HuggingFace if not cached locally)
pipe = build_foleycrafter(
    pretrained_model_name_or_path="auffusion/auffusion-full-no-adapter"
).to("cuda")

print(type(pipe))
# <class 'foleycrafter.pipelines.pipeline_controlnet.StableDiffusionControlNetPipeline'>
print(pipe.controlnet)
# ControlNetModel(...)
```

--------------------------------

### run_inference

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Performs a full batch inference loop, processing all .mp4 files in an input folder. It runs the complete pipeline, saves the generated WAV audio, and muxes the audio back into the original video using MoviePy.

```APIDOC
## run_inference

### Description
This function orchestrates the end-to-end inference process. It iterates over input video files, generates audio, saves it, and merges the audio back into the video.

### Method
```python
run_inference(config, pipe, vocoder, time_detector)
```

### Parameters
- **config** (argparse.Namespace) - Configuration object containing inference parameters.
- **pipe** (object) - The main diffusion pipeline object.
- **vocoder** (object) - The vocoder model for audio synthesis.
- **time_detector** (object) - The time detector model.

### CLI Usage Example
```bash
python inference.py \
  --input=examples/sora/ \
  --save_dir=output/sora/ \
  --seed=42 \
  --prompt="" \
  --nprompt="" \
  --semantic_scale=1.0 \
  --temporal_scale=0.2
```

### Programmatic Call Example
```python
import argparse
from inference import args_parse, build_models, run_inference

config = argparse.Namespace(
    prompt="",
    nprompt="",
    seed=42,
    semantic_scale=1.0,
    temporal_scale=0.2,
    input="examples/sora",
    ckpt="checkpoints/",
    save_dir="output/sora/",
    pretrain="auffusion/auffusion-full-no-adapter",
    device="cuda",
)

pipe, vocoder, time_detector = build_models(config)
run_inference(config, pipe, vocoder, time_detector)
```

### Outputs
- `output/sora/audio/0.wav` (16kHz mono WAV)
- `output/sora/video/0.mp4` (original video with generated audio)
```

--------------------------------

### Mel Spectrogram to Audio Reconstruction with SpectrogramConverter

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Reconstructs pydub AudioSegment objects from mel-amplitude numpy arrays using Griffin-Lim. The reconstructed audio can be exported to a WAV file.

```python
# Spectrogram → audio (approximate via Griffin-Lim)
reconstructed = converter.audio_from_spectrogram(spectrogram, apply_filters=True)
print(type(reconstructed))  # <class 'pydub.audio_segment.AudioSegment'>
reconstructed.export("reconstructed.wav", format="wav")
```

--------------------------------

### Audio to Mel Spectrogram Conversion with SpectrogramConverter

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Converts pydub AudioSegment objects to mel-amplitude numpy arrays using SpectrogramParams. Ensure the audio is set to the correct sample rate before conversion.

```python
import pydub
from foleycrafter.utils.util import SpectrogramParams, SpectrogramConverter

params = SpectrogramParams(
    sample_rate=16000,
    stereo=False,
    step_size_ms=10,
    window_duration_ms=100,
    padded_duration_ms=400,
    num_frequencies=256,
    min_frequency=0,
    max_frequency=8000,
)

converter = SpectrogramConverter(params=params, device="cpu")

# Audio → spectrogram
audio_segment = pydub.AudioSegment.from_file("examples/avsync/0.mp4").set_frame_rate(16000)
spectrogram = converter.spectrogram_from_audio(audio_segment)
print(spectrogram.shape)  # (1, 256, T) — (channels, mel_bins, time)
```

--------------------------------

### Decode Spectrogram to Waveform with HiFi-GAN

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Uses the HiFi-GAN Generator to convert a log-mel spectrogram to an audio waveform. Ensure the vocoder is loaded and the spectrogram tensor is on the correct device (e.g., CUDA). The output can be trimmed to a specific duration or number of samples.

```python
import soundfile as sf
from foleycrafter.pipelines.auffusion_pipeline import Generator

vocoder = Generator.from_pretrained("checkpoints/", subfolder="vocoder").to("cuda")

# spectrogram: (256, 1024) log-mel tensor on CUDA
audio = vocoder.inference(spectrogram, lengths=160000)[0]  # trim to 10 s @ 16kHz

# Trim to actual video duration (e.g. 4.97 s)
audio = audio[: int(duration * 16000)]
print(audio.shape)   # (79520,)  int16 samples

sf.write("output/audio/result.wav", audio, samplerate=16000)
# Writes a mono 16kHz WAV file
```

--------------------------------

### FoleyController.foley

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

A stateful controller class for the Gradio web interface. It manages model loading and inference, accepting a video file path and UI parameters to return the path of the output video with generated audio.

```APIDOC
## FoleyController.foley

### Description
This method wraps the complete model loading and inference logic for the Gradio demo. It takes an input video and various UI parameters to generate and return an output video with synthesized audio.

### Method
```python
controller.foley(input_video, prompt_textbox, negative_prompt_textbox, ip_adapter_scale, temporal_scale, sampler_dropdown, sample_step_slider, cfg_scale_slider, seed_textbox)
```

### Parameters
- **input_video** (str) - Path to the input video file.
- **prompt_textbox** (str, optional) - Text prompt for generation. Defaults to "".
- **negative_prompt_textbox** (str, optional) - Negative prompt for generation. Defaults to "".
- **ip_adapter_scale** (float) - Visual content scale (0–1). Defaults to 1.0.
- **temporal_scale** (float) - Temporal alignment scale (0–1). Defaults to 0.2.
- **sampler_dropdown** (str) - Sampler to use (e.g., "DDIM", "Euler", "PNDM"). Defaults to "DDIM".
- **sample_step_slider** (int) - Number of denoising steps (10–100). Defaults to 25.
- **cfg_scale_slider** (float) - Classifier-free guidance scale. Defaults to 7.5.
- **seed_textbox** (str) - Reproducibility seed. Defaults to "33817921".

### Request Example
```python
# Assuming controller is an instance of FoleyController
result_video_path = controller.foley(
    input_video="examples/gen3/case1.mp4",
    prompt_textbox="",                 # optional text prompt
    negative_prompt_textbox="",        # optional negative prompt
    ip_adapter_scale=1.0,             # visual content scale (0–1)
    temporal_scale=0.2,               # temporal alignment scale (0–1)
    sampler_dropdown="DDIM",          # DDIM | Euler | PNDM
    sample_step_slider=25,            # denoising steps (10–100)
    cfg_scale_slider=7.5,             # classifier-free guidance scale
    seed_textbox="33817921",          # reproducibility seed
)
print(result_video_path)
# "samples/Gradio-2024-07-01T00-00-00/sample/output.mp4"
```

### Response
- **result_video_path** (str) - Path to the output video file with generated audio.
```

--------------------------------

### BibTeX Entry for FoleyCrafter

Source: https://github.com/open-mmlab/foleycrafter/blob/main/README.md

This is the BibTeX entry for citing the FoleyCrafter paper.

```bibtex
@misc{zhang2024pia,
  title={FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds},
  author={Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, Kai Chen},
  year={2024},
  eprint={2407.01494},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}
```

--------------------------------

### Spectrogram to Diffusion Latent Conversion

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Provides functions for converting between log-mel spectrograms and diffusion model latents. `normalize_spectrogram` prepares spectrograms for VAE input by clamping, scaling, and applying a power curve. `denormalize_spectrogram` reverses this process to recover the original spectrogram scale.

```python
import torch
from foleycrafter.pipelines.auffusion_pipeline import normalize_spectrogram, denormalize_spectrogram

# Simulate a log-mel spectrogram from training data (e.g., loaded from disk)
log_mel = torch.randn(256, 1024)  # (mel_bins, time_frames)

# Normalize for diffusion model input
normalized = normalize_spectrogram(log_mel, max_value=200, min_value=1e-5, power=1.0)
print(normalized.shape)   # torch.Size([3, 256, 1024])
print(normalized.min(), normalized.max())  # approximately 0.0 and 1.0

# Denormalize back after generation
recovered = denormalize_spectrogram(normalized, max_value=200, min_value=1e-5, power=1)
print(recovered.shape)    # torch.Size([256, 1024])
# recovered ≈ log_mel (within floating-point precision)
```

--------------------------------

### Convert Diffusion Output to Log-Mel Spectrogram

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Reverses normalization applied during diffusion model training. Converts the output tensor to a log-mel spectrogram ready for vocoder decoding.

```python
from foleycrafter.pipelines.auffusion_pipeline import denormalize_spectrogram

# audio_img: torch.Tensor of shape (3, 256, 1024) from pipeline output
spectrogram = denormalize_spectrogram(
    audio_img,
    max_value=200,
    min_value=1e-5,
    power=1,
)
print(spectrogram.shape)   # torch.Size([256, 1024])
print(spectrogram.min(), spectrogram.max())  # roughly (-11.5, 5.3) log scale
```

--------------------------------

### Extract Video Frames with MoviePy

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Reads frames from an MP4 file using MoviePy and resamples them to a specified number of uniformly spaced frames. Returns the frame array and video duration. This is used to prepare temporal and semantic inputs for the FoleyCrafter pipeline. Ensure the video file exists at the specified path.

```python
import numpy as np
from foleycrafter.utils.util import read_frames_with_moviepy

frames, duration = read_frames_with_moviepy(
    video_path="examples/sora/0.mp4",
    max_frame_nums=150,
)

print(frames.shape)   # (150, H, W, 3)  — uint8 RGB
print(duration)       # e.g. 4.966666...  seconds
```

--------------------------------

### normalize_spectrogram / denormalize_spectrogram

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Utility functions for converting between log-mel spectrogram tensors and the diffusion latent space. `normalize_spectrogram` prepares a spectrogram for the VAE, while `denormalize_spectrogram` recovers the original scale after generation.

```APIDOC
## normalize_spectrogram / denormalize_spectrogram

### Description
These functions facilitate the conversion between log-mel spectrograms and the latent representation used by the diffusion model. `normalize_spectrogram` maps a log-mel spectrogram to the `[0, 1]` range expected by the VAE, and `denormalize_spectrogram` performs the inverse operation.

### Methods
```python
normalize_spectrogram(spectrogram, max_value, min_value, power)
denormalize_spectrogram(normalized_spectrogram, max_value, min_value, power)
```

### Parameters
- **spectrogram** (Tensor) - Input log-mel spectrogram tensor `(mel_bins, time_frames)`.
- **normalized_spectrogram** (Tensor) - Normalized spectrogram tensor, typically with 3 channels.
- **max_value** (float) - The maximum value in the original spectrogram scale.
- **min_value** (float) - The minimum value in the original spectrogram scale.
- **power** (float) - The power curve to apply during normalization/denormalization.

### Request Example
```python
import torch
from foleycrafter.pipelines.auffusion_pipeline import normalize_spectrogram, denormalize_spectrogram

# Simulate a log-mel spectrogram from training data (e.g., loaded from disk)
log_mel = torch.randn(256, 1024)  # (mel_bins, time_frames)

# Normalize for diffusion model input
normalized = normalize_spectrogram(log_mel, max_value=200, min_value=1e-5, power=1.0)
print(normalized.shape)   # torch.Size([3, 256, 1024])
print(normalized.min(), normalized.max())  # approximately 0.0 and 1.0

# Denormalize back after generation
recovered = denormalize_spectrogram(normalized, max_value=200, min_value=1e-5, power=1)
print(recovered.shape)    # torch.Size([256, 1024])
# recovered ≈ log_mel (within floating-point precision)
```

### Response
- **normalized** (Tensor) - The normalized spectrogram tensor suitable for VAE input.
- **recovered** (Tensor) - The denormalized spectrogram tensor, approximating the original log-mel scale.
```

--------------------------------

### Extract CLIP Image Embeddings

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Extracts semantic visual features from video frames using CLIP. It processes frames, obtains per-frame embeddings, averages them, and prepends a zero tensor for classifier-free guidance.

```python
import torch
from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection

image_processor = CLIPImageProcessor()
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "h94/IP-Adapter", subfolder="models/image_encoder"
).to("cuda")

# frames: numpy (150, H, W, 3) — subsample every 10th for CLIP
clip_frames = frames[::10]

images = image_processor(images=clip_frames, return_tensors="pt").to("cuda")
with torch.no_grad():
    image_embeddings = image_encoder(**images).image_embeds  # (N_frames, 768)

image_embeddings = torch.mean(image_embeddings, dim=0, keepdim=True).unsqueeze(0).unsqueeze(0)
# shape: (1, 1, 1, 768)

neg_image_embeddings = torch.zeros_like(image_embeddings)
image_embeddings = torch.cat([neg_image_embeddings, image_embeddings], dim=1)
# final shape for pipeline: (1, 2, 1, 768)
print(image_embeddings.shape)
```

--------------------------------

### Generator.inference

Source: https://context7.com/open-mmlab/foleycrafter/llms.txt

Decodes a log-mel spectrogram tensor into a raw audio waveform using the HiFi-GAN vocoder. This method handles numpy conversion and can optionally trim the output to a maximum number of samples.

```APIDOC
## Generator.inference

### Description
Decodes a log-mel spectrogram tensor `(mel_bins, time)` to a raw audio waveform using the HiFi-GAN vocoder. It wraps the forward pass, handles numpy conversion, and optionally trims to a maximum number of samples.

### Method
```python
Generator.inference(spectrogram, lengths=None)
```

### Parameters
- **spectrogram** (Tensor) - Input log-mel spectrogram tensor `(mel_bins, time)` on CUDA.
- **lengths** (int, optional) - Maximum number of samples to trim the output audio to. Defaults to None.

### Request Example
```python
import soundfile as sf
from foleycrafter.pipelines.auffusion_pipeline import Generator

vocoder = Generator.from_pretrained("checkpoints/", subfolder="vocoder").to("cuda")

# spectrogram: (256, 1024) log-mel tensor on CUDA
audio = vocoder.inference(spectrogram, lengths=160000)[0]  # trim to 10 s @ 16kHz

# Trim to actual video duration (e.g. 4.97 s)
audio = audio[: int(duration * 16000)]
print(audio.shape)   # (79520,)  int16 samples

sf.write("output/audio/result.wav", audio, samplerate=16000)
# Writes a mono 16kHz WAV file
```

### Response
- **audio** (Tensor) - Generated raw audio waveform.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.