### Implement Custom AudioCraft Solver

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/TRAINING.md

Example of creating a custom solver by extending `StandardSolver`. It demonstrates how to define custom attributes, build models and dataloaders, and implement training/validation steps.

```python
from . import base
from .. import optim


class MyNewSolver(base.StandardSolver):

    def __init__(self, cfg: omegaconf.DictConfig):
        super().__init__(cfg)
        # one can add custom attributes to the solver
        self.criterion = torch.nn.L1Loss()

    def best_metric(self):
        # here optionally specify which metric to use to keep track of best state
        return 'loss'

    def build_model(self):
        # here you can instantiate your models and optimization related objects
        # this method will be called by the StandardSolver init method
        self.model = ...
        # the self.cfg attribute contains the raw configuration
        self.optimizer = optim.build_optimizer(self.model.parameters(), self.cfg.optim)
        # don't forget to register the states you'd like to include in your checkpoints!
        self.register_stateful('model', 'optimizer')
        # keep the model best state based on the best value achieved at validation for the given best_metric
        self.register_best('model')
        # if you want to add EMA around the model
        self.register_ema('model')

    def build_dataloaders(self):
        # here you can instantiate your dataloaders
        # this method will be called by the StandardSolver init method
        self.dataloaders = ...

    ...

    # For both train and valid stages, the StandardSolver relies on
    # a share common_train_valid implementation that is in charge of
    # accessing the appropriate loader, iterate over the data up to
    # the specified number of updates_per_epoch, run the ``run_step``
    # function that you need to implement to specify the behavior
    # and finally update the EMA and collect the metrics properly.
    @abstractmethod
    def run_step(self, idx: int, batch: tp.Any, metrics: dict):
        """Perform one training or valid step on a given batch.
        """
        ... # provide your implementation of the solver over a batch

    def train(self):
        """Train stage.
        """
        return self.common_train_valid('train')

    def valid(self):
        """Valid stage.
        """
        return self.common_train_valid('valid')

    @abstractmethod
    def evaluate(self):
        """Evaluate stage.
        """
        ... # provide your implementation here!

    @abstractmethod
    def generate(self):
        """Generate stage.
        """
        ... # provide your implementation here!

```

--------------------------------

### Launch Compression Task Grid with Dora

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/TRAINING.md

This command initiates a compression task using a debug grid through the Dora experiment manager. It's an example of how to run jobs using Dora grids for more complex experiment setups.

```shell
dora grid compression.debug
```

--------------------------------

### Schedule MusicGen Training Jobs with Dora

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md

Schedules MusicGen training jobs using the 'dora' command-line tool. Examples are provided for text-to-music and melody-guided music generation. The `--dry_run --init` flags are used for initial setup and testing before full job scheduling. This command assumes a pre-configured environment and the 'dora' tool.

```shell
# text-to-music
dora grid musicgen.musicgen_base_32khz --dry_run --init
# melody-guided music generation
dora grid musicgen.musicgen_melody_base_32khz --dry_run --init
# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.

```

--------------------------------

### Start MOS Tool for Sample Comparison

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md

This command starts the MOS (Mean Opinion Score) tool using gunicorn, which allows easy access and comparison of generated audio samples between different models. Requires Flask and gunicorn to be installed.

```shell
gunicorn -w 4 -b 127.0.0.1:8895 -t 120 'scripts.mos:app'  --access-logfile -
```

--------------------------------

### Get Experiment Information with Dora

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/TRAINING.md

Retrieves hyper-parameters for a specific experiment (XP) identified by its signature. It can also tail logs for scheduled XPs. Be aware that overrides might appear twice, with the rightmost one being the effective value.

```shell
dora info -f 81de367c  # this will show the hyper-parameter used by a specific XP.
                       # Be careful some overrides might present twice, and the right most one
                       # will give you the right value for it.

dora info -f SIG -t    # will tail the log (if the XP has scheduled).
# if you need to access the logs of the process for rank > 0, in particular because a crash didn't happen in the main
# process, then use `dora info -f SIG` to get the main log name (finished into something like `/5037674_0_0_log.out`)
# and worker K can be accessed as `/5037674_0_{K}_log.out`.
# This is only for scheduled jobs, for local distributed runs with `-d`, then you should go into the XP folder,
# and look for `worker_{K}.log` logs.
```

--------------------------------

### AudioCraft Dataset Configuration Example

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/DATASETS.md

Example of a datasource configuration file for an AudioCraft dataset. It specifies maximum sample rate and channels, and lists the paths to manifest files for different training stages (train, valid, evaluate, generate).

```yaml
# @package __global__

datasource:
  max_sample_rate: 44100
  max_channels: 2

  train: egs/example
  valid: egs/example
  evaluate: egs/example
  generate: egs/example
```

--------------------------------

### Launch Compression Task with Encodec using Dora

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/TRAINING.md

This command launches a compression task using the lightweight encodec model via the Dora experiment manager. It demonstrates a basic use case for running local jobs.

```shell
dora run solver=compression/debug
```

--------------------------------

### Install AudioCraft and Dependencies (Shell)

Source: https://github.com/facebookresearch/audiocraft/blob/main/README.md

Installs the AudioCraft library and its core dependencies like PyTorch and ffmpeg. It covers stable releases, bleeding-edge versions, local installations, and optional dependencies for watermarking models. It also includes commands for installing ffmpeg via apt-get or conda.

```shell
python -m pip install 'torch==2.1.0'
python -m pip install setuptools wheel
python -m pip install -U audiocraft  # stable release
python -m pip install -U git+https://git@github.com/facebookresearch/audiocraft#egg=audiocraft  # bleeding edge
python -m pip install -e .  # or if you cloned the repo locally (mandatory if you want to train).
python -m pip install -e '.[wm]'  # if you want to train a watermarking model
sudo apt-get install ffmpeg
conda install "ffmpeg<5" -c conda-forge
```

--------------------------------

### Activate and Configure ViSQOL Metric

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/METRICS.md

Activates the ViSQOL (Virtual Speech Quality Objective Listener) metric and specifies the path to its binary installation. ViSQOL requires a Python wrapper and an external binary.

```shell
# The first parameter activates ViSQOL computation, the second specifies the path to its library.
dora run <...> evaluate.metrics.visqol=true metrics.visqol.bin=<path_to_visqol>
```

--------------------------------

### Audio Compression API Example

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MBD.md

Example demonstrating how to use MultiBand Diffusion for audio compression and comparison with EnCodec.

```APIDOC
## Audio Compression API Example

### Description
This example shows how to use the `MultiBandDiffusion` model for audio compression and how to compare its output with the `EnCodec` model. It allows you to compress audio using specified bandwidths.

### Method
Python Script

### Endpoint
N/A (Local Script Execution)

### Parameters
- **bandwidth** (float) - Required - The desired bandwidth for compression (e.g., 1.5, 3.0, 6.0).
- **somepath** (str) - Required - The file path to the input audio.

### Request Example
```python
import torch
from audiocraft.models import MultiBandDiffusion
from encodec import EncodecModel
from audiocraft.data.audio import audio_read, audio_write

# Specify bandwidth and load models
bandwidth = 3.0  # 1.5, 3.0, 6.0
mbd = MultiBandDiffusion.get_mbd_24khz(bw=bandwidth)
encodec = EncodecModel.encodec_model_24khz()

somepath = '' # Replace with your audio file path
wav, sr = audio_read(somepath)

# Compress audio using EnCodec and MultiBandDiffusion
with torch.no_grad():
    compressed_encodec = encodec(wav)
    compressed_diffusion = mbd.regenerate(wav, sample_rate=sr)

# Save compressed audio
audio_write('sample_encodec', compressed_encodec.squeeze(0).cpu(), mbd.sample_rate, strategy="loudness", loudness_compressor=True)
audio_write('sample_diffusion', compressed_diffusion.squeeze(0).cpu(), mbd.sample_rate, strategy="loudness", loudness_compressor=True)
```

### Response
#### Success Response (N/A - Local Script)
- **compressed_encodec** (torch.Tensor) - Compressed audio using EnCodec.
- **compressed_diffusion** (torch.Tensor) - Compressed audio using MultiBand Diffusion.

#### Response Example
N/A (Audio files are saved to disk)
```

--------------------------------

### Training Custom AudioCraft Models with Hydra

Source: https://context7.com/facebookresearch/audiocraft/llms.txt

This section outlines the process for training custom AudioCraft models using the solver framework and Hydra for configuration management. It includes an example YAML configuration file and command-line instructions for launching training and monitoring with TensorBoard.

```yaml
# Training is configured via Hydra YAML files, not Python API
# Example: Train custom MusicGen model

# 1. Create config file: config/solver/custom_musicgen.yaml
solver: musicgen
sample_rate: 32000
channels: 1
compression_model_checkpoint: //pretrained/facebook/encodec_32khz

dataset:
  batch_size: 64
  segment_duration: 30
  num_workers: 8
  train: /path/to/train_dataset
  valid: /path/to/valid_dataset

optim:
  optimizer: adamw
  lr: 0.0001
  weight_decay: 0.01
  epochs: 100

model:
  lm_model: transformer_lm
  transformer_lm:
    dim: 1024
    num_heads: 16
    num_layers: 24
    causal: true
    cross_attention: true

conditioners:
    description:
      model: t5
      t5_model_name: t5-base

# 2. Launch training via command line
# python -m audiocraft.train \
#     solver=custom_musicgen \
#     dset=audio/default \
#     model/lm/model_scale=medium \
#     continue_from=/path/to/checkpoint.pt  # Optional resume

# 3. Monitoring with tensorboard
# tensorboard --logdir outputs/
```

```python
# Example Python training script (advanced)
from audiocraft.solvers import MusicGenSolver
from omegaconf import OmegaConf
import torch

# Load config
cfg = OmegaConf.load('config/solver/custom_musicgen.yaml')

# Initialize solver
solver = MusicGenSolver(cfg)

# Training loop (simplified)
for epoch in range(cfg.optim.epochs):
    # Train epoch
    train_metrics = solver.run_epoch()
    print(f"Epoch {epoch}: loss={train_metrics['loss']:.4f}")

    # Validation
    if epoch % 5 == 0:
        with torch.no_grad():
            val_metrics = solver.run_valid()
        print(f"Validation: loss={val_metrics['loss']:.4f}")

        # Save checkpoint
        solver.save_checkpoint(epoch)

# Load trained checkpoint for inference
from audiocraft.models import MusicGen
custom_model = MusicGen.get_pretrained('outputs/my_checkpoint.pt')
```

--------------------------------

### Run Experiments with Dora

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/TRAINING.md

Executes an experiment using hyper-parameters from a specified XP signature. Supports distributed runs using all available GPUs with the `-d` flag. It allows overriding specific hyper-parameters from the base XP signature, creating a new XP with a unique signature. The `--clear` flag can be used to ignore previous checkpoints and restart from scratch.

```shell
dora run -d -f 81de367c   # run an XP with the hyper-parameters from XP 81de367c.
                          # `-d` is for distributed, it will use all available GPUs.

dora run -d -f 81de367c dataset.batch_size=32  # start from the config of XP 81de367c but change some hyper-params.
                                                # This will give you a new XP with a new signature (e.g. 3fe9c332).

dora run [-f BASE_SIG] [ARGS] --clear
# The following will delete the folder and checkpoint for a single XP,
# and then run it afresh.
```

--------------------------------

### Run MusicGen Training Grid

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN_STYLE.md

This command schedules jobs to reproduce the training of MusicGen-Style. Remove `--dry_run --init` flags to activate job scheduling after setup.

```shell
# text-and-style-to-music
dora grid musicgen.musicgen_style_32khz --dry_run --init

# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.
```

--------------------------------

### Manage Grids with Dora

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/TRAINING.md

Enables running experiments defined in a grid configuration file. The `dora grid` command can execute a grid, display its configuration with `--dry_run`, and initialize the Dora experiments database with `--init`. It also supports clearing all XPs associated with a grid and rescheduling them from scratch using the `--clear` flag.

```shell
# Run a dummy grid located at `audiocraft/grids/my_grid_folder/my_grid_name.py`
dora grid my_grid_folder.my_grid_name
# The following will simply display the grid and also initialize the Dora experiments database.
# You can then simply refer to a config using its signature (e.g. as `dora run -f SIG`).
dora grid my_grid_folder.my_grid_name --dry_run --init

# This will cancel all the XPs and delete their folder and checkpoints.
# It will then reschedule them starting from scratch.
dora grid my_grid_folder.my_grid_name --clear
```

--------------------------------

### Install Hugging Face Transformers library

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md

Installs the latest version of the Hugging Face Transformers library from its GitHub repository. This is a prerequisite for using the MusicGen model with the library.

```shell
pip install git+https://github.com/huggingface/transformers.git
```

--------------------------------

### Style-to-Music Generation with Text and Melody Conditioning (Python)

Source: https://github.com/facebookresearch/audiocraft/blob/main/demos/musicgen_style_demo.ipynb

Generates an 8-second music clip conditioned on both text descriptions and a melody input. This example utilizes double CFG, which is necessary for text-and-style conditioning, with a specified beta value. Dependencies include `torchaudio` and `audiocraft.utils.notebook`.

```python
import torchaudio
from audiocraft.utils.notebook import display_audio

model.set_generation_params(
    duration=8, # generate 8 seconds, can go up to 30
    use_sampling=True, 
    top_k=250,
    cfg_coef=3., # Classifier Free Guidance coefficient 
    cfg_coef_beta=5., # double CFG is necessary for text-and-style conditioning
                   # Beta in the double CFG formula. between 1 and 9. When set to 1 
                   # it is equivalent to normal CFG. 
)

model.set_style_conditioner_params(
    eval_q=1, # integer between 1 and 6
              # eval_q is the level of quantization that passes
              # through the conditioner. When low, the models adheres less to the 
              # audio conditioning
    excerpt_length=3., # the length in seconds that is taken by the model in the provided excerpt
    )

melody_waveform, sr = torchaudio.load("../assets/electronic.mp3")
melody_waveform = melody_waveform.unsqueeze(0).repeat(3, 1, 1)

descriptions = ["8-bit old video game music", "Chill lofi remix", "80s New wave with synthesizer"]

output = model.generate_with_chroma(
    descriptions=descriptions,
    melody_wavs=melody_waveform,
    melody_sample_rate=sr,
    progress=True, return_tokens=True
)
display_audio(output[0], sample_rate=32000)
if USE_DIFFUSION_DECODER:
    out_diffusion = mbd.tokens_to_wav(output[1])
    display_audio(out_diffusion, sample_rate=32000)
```

--------------------------------

### AudioGen Training Command Example (Shell)

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/AUDIOGEN.md

This shell command initiates the training process for AudioGen using the 'dora' command-line tool and a specified configuration. It's designed for text-to-sound generation tasks.

```shell
# text-to-sound
dora grid audiogen.audiogen_base_16khz
```

--------------------------------

### Clone Repository and Set Up Virtual Environment

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/JASCO.md

Commands to clone the Deepsalience repository, navigate into it, create a conda virtual environment with Python 3.7, activate it, and install the required dependencies.

```bash
git clone git@github.com:lonzi/ismir2017-deepsalience.git forked_deepsalience_repo
cd forked_deepsalience_repo
conda create --name deep_salience python=3.7
conda activate deep_salience
pip install -r requirements.txt
```

--------------------------------

### Melody + Drums + Chords Conditioning Example - Python

Source: https://github.com/facebookresearch/audiocraft/blob/main/jasco_demo.ipynb

An advanced inference example demonstrating music generation conditioned on melody, drums, and chords. It involves loading audio, processing melody prompts, separating drum stems using Demucs, and finally generating music with `model.generate_music`. Includes plotting utilities for visualization.

```python
%matplotlib inline
import torchaudio 
from audiocraft.models import JASCO
from demucs import pretrained
from demucs.apply import apply_model
from demucs.audio import convert_audio
import torch
from audiocraft.utils.notebook import display_audio
import matplotlib.pyplot as plt

# --------------------------
# First, choose file to load
# --------------------------
fnames = ['salience_1', 'salience_2']
chords = [
    [('N',  0.0), ('Eb7',  1.088000000), ('C#',  4.352000000), ('D',  4.864000000), ('Dm7',  6.720000000), ('G7',  8.256000000), ('Am7b5/G',  9.152000000)],  # for salience 1
    [('N',  0.0), ('C',  0.320000000), ('Dm7',  3.456000000), ('Am',  4.608000000), ('F',  8.320000000), ('C',  9.216000000)]  # for salience 2
]
file_idx = 1  # either 0 or 1


# ------------------------------------
# display audio, melody map and chords
# ------------------------------------
def plot_chromagram(tensor):
    # Check if tensor is a PyTorch tensor
    if not torch.is_tensor(tensor):
        raise ValueError('Input should be a PyTorch tensor')
    tensor = tensor.numpy().T # C, T
    plt.figure(figsize=(20, 20))
    plt.imshow(tensor, cmap='binary', interpolation='nearest', origin='lower')
    plt.show()

# load salience and display the corresponding wav
melody_prompt_wav, melody_prompt_sr = torchaudio.load(f"./assets/{fnames[file_idx]}.wav")
print("Source melody:")
display_audio(melody_prompt_wav, sample_rate=melody_prompt_sr)
melody =  torch.load(f"./assets/{fnames[file_idx]}.th", weights_only=True)
plot_chromagram(melody)
print("Chords:")
print(chords[file_idx])

# --------------------------------------------------
# use demucs to seperate the drums stem from src mix
# --------------------------------------------------
def _get_drums_stem(wav: torch.Tensor, sample_rate: int) -> torch.Tensor:
    """Get parts of the wav that holds the drums, extracting the main stems from the wav."""
    demucs_model = pretrained.get_model('htdemucs').to('cuda')
    wav = convert_audio(
        wav, sample_rate, demucs_model.samplerate, demucs_model.audio_channels)  # type: ignore
    stems = apply_model(demucs_model, wav.cuda().unsqueeze(0), device='cuda').squeeze(0)
    drum_stem = stems[demucs_model.sources.index('drums')]  # extract relevant stems for drums conditioning
    return convert_audio(drum_stem.cpu(), demucs_model.samplerate, sample_rate, 1)  # type: ignore
drums_wav = _get_drums_stem(melody_prompt_wav, melody_prompt_sr)
print("Separated drums:")
display_audio(drums_wav, sample_rate=melody_prompt_sr)

# ----------------------------------
# Generate using the loaded controls
# ----------------------------------
# these are free-form texts written randomly
texts = [
    '90s rock with heavy drums and hammond',
    '80s pop with groovy synth bass and drum machine',
    'folk song with leading accordion',
]

print("Generating...")
```

--------------------------------

### AudioGen Evaluation Configuration Examples (Shell)

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/AUDIOGEN.md

These shell commands demonstrate how to activate and configure the evaluation stage for AudioGen, including computing metrics like cross-entropy and perplexity. They utilize the 'dora' tool with specific configuration paths.

```shell
# using the configuration
dora run solver=audiogen/debug solver/audiogen/evaluation=objective_eval
# specifying each of the fields, e.g. to activate KL computation
dora run solver=audiogen/debug evaluate.metrics.kld=true
```

--------------------------------

### Install and Activate Text Consistency Metric (CLAP)

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/METRICS.md

Installs the laion_clap library and activates the text consistency metric using the CLAP model. This metric relies on a pre-trained Contrastive Language-Audio Pretraining model and requires its checkpoint.

```shell
pip install laion_clap
dora run ... evaluate.metrics.text_consistency=true metrics.text_consistency.model=clap
```

--------------------------------

### Run MusicGen Training with Custom Audio Tokenizers using Dora

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md

Configures and runs MusicGen training using the 'dora' tool with different audio tokenizers. Examples show how to use pre-trained EnCodec models, DAC, or custom models by specifying the `compression_model_checkpoint` and other relevant parameters like `transformer_lm.n_q` and `transformer_lm.card`.

```bash
# Using the 32kHz EnCodec trained on music
dora run solver=musicgen/debug \
    compression_model_checkpoint=//pretrained/facebook/encodec_32khz \
    transformer_lm.n_q=4 transformer_lm.card=2048

# Using DAC
dora run solver=musicgen/debug \
    compression_model_checkpoint=//pretrained/dac_44khz \
    transformer_lm.n_q=9 transformer_lm.card=1024 \
    'codebooks_pattern.delay.delays=[0,1,2,3,4,5,6,7,8]'

# Using your own model after export (see ENCODEC.md)
dora run solver=musicgen/debug \
    compression_model_checkpoint=//pretrained//checkpoints/my_audio_lm/compression_state_dict.bin \
    transformer_lm.n_q=... transformer_lm.card=...

```

--------------------------------

### Fine-tune Existing JASCO Models using Dora

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/JASCO.md

Demonstrates how to fine-tune existing JASCO models using the 'dora run' command. It shows examples of continuing from a pre-trained JASCO model or a previously trained model identified by its Dora signature.

```bash
# Using pretrained JASCO model.
dora run solver=jasco/chords_drums model/lm/model_scale=small continue_from=//pretrained/facebook/jasco-chords-drums-400M conditioner=jasco_chords_drums

# Using another model you already trained with a Dora signature SIG.
dora run solver=jasco/chords_drums model/lm/model_scale=small continue_from=//sig/SIG conditioner=jasco_chords_drums
```

--------------------------------

### Get Evaluation Solver Instance

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/JASCO.md

Instantiates an evaluation solver for the Jasco model. This Python snippet retrieves a pre-configured solver instance for evaluation purposes, specifying the device and batch size. The resulting solver object provides access to the model and dataloaders.

```python
from audiocraft.solvers.jasco import JascoSolver

solver = JascoSolver.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)
solver.model
solver.dataloaders
```

--------------------------------

### Initialize MAGNeT for Music Generation

Source: https://github.com/facebookresearch/audiocraft/blob/main/demos/magnet_demo.ipynb

Initializes the MAGNeT model for music generation using a pre-trained variant like 'facebook/magnet-small-10secs'. This step requires the audiocraft library and loads the specified model.

```python
from audiocraft.models import MAGNeT

model = MAGNeT.get_pretrained('facebook/magnet-small-10secs')
```

--------------------------------

### MusicGen API Example

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MBD.md

Example demonstrating the use of MultiBand Diffusion with the MusicGen API for generating unconditional and conditional audio samples, including melody conditioning.

```APIDOC
## MusicGen API Example

### Description
This example demonstrates how to use the `MusicGen` and `MultiBandDiffusion` models to generate unconditional and conditional audio samples. It covers generating audio from text descriptions and generating audio conditioned on a melody from an existing audio file.

### Method
Python Script

### Endpoint
N/A (Local Script Execution)

### Parameters
N/A (Model Loading and Generation)

### Request Example
```python
import torchaudio
from audiocraft.models import MusicGen, MultiBandDiffusion
from audiocraft.data.audio import audio_write

# Load pre-trained models
model = MusicGen.get_pretrained('facebook/musicgen-melody')
mbd = MultiBandDiffusion.get_mbd_musicgen()

# Set generation parameters
model.set_generation_params(duration=8)  # generate 8 seconds.

# Unconditional generation
wav, tokens = model.generate_unconditional(4, return_tokens=True)    # generates 4 unconditional audio samples and keep the tokens for MBD generation
descriptions = ['happy rock', 'energetic EDM', 'sad jazz']
wav_diffusion = mbd.tokens_to_wav(tokens)

# Conditional generation with text descriptions
wav, tokens = model.generate(descriptions, return_tokens=True)  # generates 3 samples and keep the tokens.
wav_diffusion = mbd.tokens_to_wav(tokens)

# Conditional generation with melody
melody, sr = torchaudio.load('./assets/bach.mp3')
# Generates using the melody from the given audio and the provided descriptions, returns audio and audio tokens.
wav, tokens = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr, return_tokens=True)
wav_diffusion = mbd.tokens_to_wav(tokens)

# Save generated audio
for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav and {idx}_diffusion.wav, with loudness normalization at -14 db LUFS for comparing the methods.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
    audio_write(f'{idx}_diffusion', wav_diffusion[idx].cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```

### Response
#### Success Response (N/A - Local Script)
- **wav** (torch.Tensor) - Generated audio waveform(s).
- **tokens** (torch.Tensor) - Intermediate tokens generated by the model.

#### Response Example
N/A (Audio files are saved to disk)
```

--------------------------------

### Run Solver with Jasco Configuration

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/JASCO.md

Launches the Jasco solver with specified model and conditioning parameters. This command-line instruction assumes a specific project structure and available checkpoints. Ensure that the 'conditioner' and 'model/lm/model_scale' parameters are compatible with the fine-tuning process.

```bash
dora run solver=jasco/chords_drums model/lm/model_scale=small conditioner=jasco_chords_drums continue_from=/checkpoints/my_other_xp/checkpoint.th
```

--------------------------------

### Install and Activate Kullback-Leibler Divergence (KLD) Metric

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/METRICS.md

Installs the PaSST classifier library and activates the Kullback-Leibler Divergence metric using the specified model (e.g., 'passt'). KLD is computed over probabilities from an audio classifier.

```shell
pip install 'git+https://github.com/kkoutini/passt_hear21@0.0.19#egg=hear21passt'
dora run <...> evaluate.metrics.kld=true metrics.kld.model=passt
```

--------------------------------

### Fine-tune MusicGen with Pretrained Model

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN_STYLE.md

Initializes a new MusicGen model using a pretrained MusicGen-Style model. Requires specifying the solver, model scale, and conditioner. The `continue_from` argument points to the pretrained model.

```bash
# Using pretrained MusicGen-Style model.
dora run solver=musicgen/musicgen_style_32khz model/lm/model_scale=medium continue_from=//pretrained/facebook/musicgen-style conditioner=style2music
```

--------------------------------

### Initialize MusicGen-Style Model

Source: https://github.com/facebookresearch/audiocraft/blob/main/demos/musicgen_style_demo.ipynb

Loads the pre-trained MusicGen model and optionally the MultiBandDiffusion decoder. This step is crucial for setting up the model for music generation.

```python
from audiocraft.models import MusicGen
from audiocraft.models import MultiBandDiffusion

USE_DIFFUSION_DECODER = False

model = MusicGen.get_pretrained('facebook/musicgen-style')
if USE_DIFFUSION_DECODER:
    mbd = MultiBandDiffusion.get_mbd_musicgen()
```

--------------------------------

### Run MusicGen Debug Solver

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md

This command initiates the MusicGen debug solver. It requires specifying the compression model checkpoint and parameters for the transformer language model, such as n_q (number of quantization levels) and card (codebook cardinality). Users are responsible for setting appropriate values for these parameters.

```bash
dora run solver=musicgen/debug \
    compression_model_checkpoint=//sig/SIG \
    transformer_lm.n_q=... transformer_lm.card=...
```

--------------------------------

### Set AUDIOCRAFT_TEAM Environment Variable (Shell)

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/TRAINING.md

Configures the AudioCraft team setting for the current Conda environment. This is essential for cluster-specific configurations and path mappings, defaulting to 'default' if not set.

```shell
conda env config vars set AUDIOCRAFT_TEAM=default
```

--------------------------------

### Initialize JASCO Model with Chord Mapping

Source: https://github.com/facebookresearch/audiocraft/blob/main/demos/jasco_demo.ipynb

Initializes the JASCO model, specifying a pre-trained model checkpoint and the path to a chord mapping file. This is the first step before configuring or using the model for music generation.

```python
import os 
from audiocraft.models import JASCO

model = JASCO.get_pretrained('facebook/jasco-chords-drums-melody-400M', chords_mapping_path='../assets/chord_to_index_mapping.pkl')
```

--------------------------------

### Load Evaluation Solver for MusicGen

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md

This Python snippet shows how to get an evaluation solver for MusicGen, loading the latest trained model. It specifies the device and batch size for the solver.

```python
from audiocraft.solvers.musicgen import MusicGenSolver

solver = MusicGenSolver.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)
solver.model
solver.dataloaders
```

--------------------------------

### MAGNeT: Audio Continuation with Prompt

Source: https://context7.com/facebookresearch/audiocraft/llms.txt

Demonstrates how to use the MAGNeT model to continue an audio prompt. It requires the prompt audio file and its sample rate, along with a textual description for continuation.

```python
import torchaudio

prompt_wav, sr = torchaudio.load('prompt.wav')
wav_continued = model.generate_continuation(
    prompt=prompt_wav,
    prompt_sample_rate=sr,
    descriptions=['continuing the musical theme']
)
```

--------------------------------

### Export AUDIOCRAFT_TEAM Environment Variable (Shell)

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/TRAINING.md

Sets the AudioCraft team configuration persistently in the user's bashrc file. This ensures the 'default' team setting is applied across all sessions.

```shell
export AUDIOCRAFT_TEAM=default
```

--------------------------------

### AudioGen Text-to-Sound Generation API Example (Python)

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/AUDIOGEN.md

This Python snippet demonstrates how to use the AudioGen API to generate audio from text descriptions. It requires the torchaudio and audiocraft libraries. The output is saved as WAV files.

```python
import torchaudio
from audiocraft.models import AudioGen
from audiocraft.data.audio import audio_write

model = AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=5)  # generate 5 seconds.
descriptions = ['dog barking', 'sirene of an emergency vehicle', 'footsteps in a corridor']
wav = model.generate(descriptions)  # generates 3 samples.

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```

--------------------------------

### Get Pretrained EnCodec Models (Python)

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/ENCODEC.md

Python code for directly retrieving pre-trained EnCodec models using the `CompressionModel.get_pretrained` method. This method does not require the `//pretrained/` prefix for model names.

```python
from audiocraft.models import CompressionModel

# Here do not put the `//pretrained/` prefix!
model = CompressionModel.get_pretrained('facebook/encodec_32khz')
model = CompressionModel.get_pretrained('dac_44khz')
```

--------------------------------

### Initialize MusicGen Model

Source: https://github.com/facebookresearch/audiocraft/blob/main/demos/musicgen_demo.ipynb

Initializes the MusicGen model, allowing selection from different pre-trained variants. It supports optional integration with a diffusion decoder. Dependencies include 'audiocraft.models.MusicGen' and 'audiocraft.models.MultiBandDiffusion'.

```python
from audiocraft.models import MusicGen
from audiocraft.models import MultiBandDiffusion

USE_DIFFUSION_DECODER = False
# Using small model, better results would be obtained with `medium` or `large`.
model = MusicGen.get_pretrained('facebook/musicgen-small')
if USE_DIFFUSION_DECODER:
    mbd = MultiBandDiffusion.get_mbd_musicgen()
```

--------------------------------

### Classifier-Free Guidance (CFG) Options

Source: https://context7.com/facebookresearch/audiocraft/llms.txt

This section covers different configurations for classifier-free guidance, including double CFG for melody conditioning and two-step CFG for more accurate padding. It shows how to adjust `cfg_coef`, `cfg_coef_beta`, and `two_step_cfg` parameters.

```python
# Double classifier-free guidance (melody conditioning)
model.set_generation_params(
    cfg_coef=3.0,               # Standard text CFG
    cfg_coef_beta=3.0,          # Melody CFG (pushes melody more)
    two_step_cfg=False          # Single-pass CFG (faster)
)

# Two-step CFG (more accurate padding)
model.set_generation_params(
    cfg_coef=3.0,
    two_step_cfg=True           # Separate forward passes
)
```

--------------------------------

### JavaScript Audio Playback Control

Source: https://github.com/facebookresearch/audiocraft/blob/main/scripts/templates/survey.html

This JavaScript code manages audio playback for multiple audio elements on the page. It ensures that only one audio element plays at a time by pausing other currently playing elements when a new one starts. This prevents overlapping audio.

```javascript
function setupCallback(elem, elems) {
  elem.addEventListener("play", function () {
    for (var other of elems) {
      if (other !== elem) {
        other.pause();
      }
    }
  });
}

document.addEventListener('DOMContentLoaded', function () {
  var elems = document.body.getElementsByTagName("audio");
  for (var elem of elems) {
    setupCallback(elem, elems);
  }
});
```

--------------------------------

### Load Audio Files with Seek and Duration Control (Python)

Source: https://context7.com/facebookresearch/audiocraft/llms.txt

Illustrates how to read audio files from disk using `audio_read` and `audio_info`. It covers retrieving file metadata, reading entire files, extracting specific segments using `seek_time` and `duration`, and padding audio to ensure a fixed duration. Batch loading for multiple files is also demonstrated.

```python
from audiocraft.data.audio import audio_read, audio_info
from audiocraft.data.audio_utils import convert_audio
import torch

# Get audio file information (fast, doesn't load audio)
info = audio_info('music.mp3')
print(f"Sample rate: {info.sample_rate}")
print(f"Duration: {info.duration:.2f}s")
print(f"Channels: {info.channels}")

# Read entire audio file
wav, sr = audio_read('music.mp3')
# wav shape: [C, T] where C=channels, T=samples
print(f"Loaded: {wav.shape} at {sr}Hz")

# Read specific portion (seek + duration)
wav, sr = audio_read(
    'music.mp3',
    seek_time=30.0,         # Start at 30 seconds
    duration=10.0           # Read 10 seconds
)
# Returns 10s of audio starting from 30s mark

# Read with padding (ensures exact duration)
wav, sr = audio_read(
    'music.mp3',
    duration=15.0,
    pad=True                # Pad with zeros if file is shorter
)
# wav will be exactly 15s long (padded if needed)

# Efficient batch loading for multiple files
audio_files = ['track1.mp3', 'track2.wav', 'track3.ogg']
batch = []
for file in audio_files:
    wav, sr = audio_read(file, duration=10.0, pad=True)
    wav = convert_audio(wav, sr, 32000, 1)  # Standardize
    batch.append(wav)

batch_tensor = torch.stack(batch)  # [B, C, T]
print(f"Batch shape: {batch_tensor.shape}")
```

--------------------------------

### Generate Music with JASCO Multi-modal Conditioning

Source: https://context7.com/facebookresearch/audiocraft/llms.txt

Provides examples of generating music using the JASCO model with various symbolic controls, including chords, drums, and melody. It shows how to save the generated audio and generate music with different combinations of conditioning.

```python
from audiocraft.models import JASCO
from audiocraft.data.audio import audio_write
import torch
import torchaudio

model = JASCO.get_pretrained(
    'facebook/jasco-chords-drums-melody-1B',
    chords_mapping_path='assets/chord_to_index_mapping.pkl'
)
model.set_generation_params(cfg_coef_all=5.0)

# Define chord progression (chord_name, start_time_in_seconds)
chords = [
    ('C', 0.0),
    ('F', 2.0),
    ('G', 4.0),
    ('Am', 6.0),
    ('C', 8.0)
]

# Load drum track (optional)
drum_wav, drum_sr = torchaudio.load('drums.wav')
from audiocraft.data.audio_utils import convert_audio
drum_wav = convert_audio(drum_wav, drum_sr, 32000, 1)  # Convert to 32kHz mono

# Load melody (optional)
melody_wav, melody_sr = torchaudio.load('melody.wav')
melody_wav = convert_audio(melody_wav, melody_sr, 32000, 1)

# Generate with multi-modal conditioning
descriptions = ['upbeat rock song']
wav = model.generate_music(
    descriptions=descriptions,
    chords=chords,              # List of (chord, time) tuples
    drums=drum_wav,             # Drum track tensor [B, C, T]
    melody=melody_wav,          # Melody tensor [B, C, T]
    progress=True
)
# Output: 10 seconds of music following all constraints

audio_write('jasco_output', wav[0].cpu(), model.sample_rate)

# Generate with only chords and text
wav_chords_only = model.generate_music(
    descriptions=['jazz ballad'],
    chords=[('Dm7', 0.0), ('G7', 2.5), ('Cmaj7', 5.0), ('Am7', 7.5)],
    drums=None,
    melody=None
)

# Generate with only drums
wav_drums_only = model.generate_music(
    descriptions=['electronic dance music'],
    chords=None,
    drums=drum_wav,
    melody=None
)
```

--------------------------------

### Initialize JASCO Model

Source: https://github.com/facebookresearch/audiocraft/blob/main/jasco_demo.ipynb

Initializes the JASCO model, specifying which pre-trained model to use and providing a path to chord mapping. It imports necessary libraries like 'os' and 'JASCO' from 'audiocraft.models'.

```python
import os 
from audiocraft.models import JASCO

chords_mapping_path = os.path.abspath('./assets/chord_to_index_mapping.pkl')
model = JASCO.get_pretrained('facebook/jasco-chords-drums-1B', chords_mapping_path='./assets/chord_to_index_mapping.pkl')
```

--------------------------------

### Convert Audio Sample Rate and Channels

Source: https://context7.com/facebookresearch/audiocraft/llms.txt

Provides utility functions to convert audio files to specific sample rates and channel configurations (mono/stereo) required by different Audiocraft models. It includes examples for MusicGen, AudioGen, and AudioSeal, as well as general conversions.

```python
from audiocraft.data.audio_utils import convert_audio
import torchaudio

# Load audio (e.g., stereo 44.1kHz)
wav, sr = torchaudio.load('audio.mp3')

# Convert to mono 32kHz for MusicGen
for_musicgen = convert_audio(wav, sr, 32000, 1)

# Convert to mono 16kHz for AudioGen and AudioSeal
for_audiogen = convert_audio(wav, sr, 16000, 1)
for_audioseal = convert_audio(wav, sr, 16000, 1)

# Example: Stereo to mono (average channels)
stereo_wav, sr = torchaudio.load('stereo.wav')
mono = convert_audio(stereo_wav, sr, sr, 1)

# Example: Mono to stereo (duplicate channel)
mono_wav, sr = torchaudio.load('mono.wav')
stereo = convert_audio(mono_wav, sr, sr, 2)

# Example: Resample without changing channels
resampled = convert_audio(wav, 44100, 16000, wav.shape[0])
```

--------------------------------

### MusicGen API Example with MultiBand Diffusion

Source: https://github.com/facebookresearch/audiocraft/blob/main/docs/MBD.md

Demonstrates using the MusicGen API to generate audio, both unconditionally and based on descriptions or a melody, and then converting the generated tokens to waveform audio using MultiBand Diffusion. It includes saving the generated audio with loudness normalization.

```python
import torchaudio
from audiocraft.models import MusicGen, MultiBandDiffusion
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('facebook/musicgen-melody')
mbd = MultiBandDiffusion.get_mbd_musicgen()
model.set_generation_params(duration=8)  # generate 8 seconds.
wav, tokens = model.generate_unconditional(4, return_tokens=True)    # generates 4 unconditional audio samples and keep the tokens for MBD generation
descriptions = ['happy rock', 'energetic EDM', 'sad jazz']
wav_diffusion = mbd.tokens_to_wav(tokens)
wav, tokens = model.generate(descriptions, return_tokens=True)  # generates 3 samples and keep the tokens.
wav_diffusion = mbd.tokens_to_wav(tokens)
melody, sr = torchaudio.load('./assets/bach.mp3')
# Generates using the melody from the given audio and the provided descriptions, returns audio and audio tokens.
wav, tokens = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr, return_tokens=True)
wav_diffusion = mbd.tokens_to_wav(tokens)

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav and {idx}_diffusion.wav, with loudness normalization at -14 db LUFS for comparing the methods.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
    audio_write(f'{idx}_diffusion', wav_diffusion[idx].cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```