Music Source Separation Training

https://github.com/zfturbo/music-source-separation-training
Admin
A comprehensive training framework for music source separation models supporting multiple...
Tokens:15,427
Snippets:161
Trust Score:9.3
License:MIT
Update:6 months ago
Show doc for...
Tokens:
Raw
### BSRoformer Training Configuration

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/bs_roformer_info.md

Example configuration for BSRoformer training, including audio, model, and training parameters. Adjust `batch_size` based on GPU memory and the provided table.

```yaml
audio:
  chunk_size: 131584
  dim_f: 1024
  dim_t: 515
  hop_length: 512
  n_fft: 2048
  num_channels: 2
  sample_rate: 44100
  min_mean_abs: 0.000

model:
  dim: 384
  depth: 12
  stereo: true
  num_stems: 1
  time_transformer_depth: 1
  freq_transformer_depth: 1
  linear_transformer_depth: 0
  freqs_per_bands: !!python/tuple
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 2
    - 4
    - 4
    - 4
    - 4
    - 4
    - 4
    - 4
    - 4
    - 4
    - 4
    - 4
    - 4
    - 12
    - 12
    - 12
    - 12
    - 12
    - 12
    - 12
    - 12
    - 24
    - 24
    - 24
    - 24
    - 24
    - 24
    - 24
    - 24
    - 48
    - 48
    - 48
    - 48
    - 48
    - 48
    - 48
    - 48
    - 128
    - 129
  dim_head: 64
  heads: 8
  attn_dropout: 0.1
  ff_dropout: 0.1
  flash_attn: false
  dim_freqs_in: 1025
  stft_n_fft: 2048
  stft_hop_length: 512
  stft_win_length: 2048
  stft_normalized: false
  mask_estimator_depth: 2
  multi_stft_resolution_loss_weight: 1.0
  multi_stft_resolutions_window_sizes: !!python/tuple
  - 4096
  - 2048
  - 1024
  - 512
  - 256
  multi_stft_hop_size: 147
  multi_stft_normalized: False

training:
  batch_size: 1
  gradient_accumulation_steps: 1
  grad_clip: 0
  instruments:
  - vocals
  - other
  lr: 3.0e-05
  patience: 2
  reduce_factor: 0.95
  target_instrument: vocals
  num_epochs: 1000
  num_steps: 1000
  q: 0.95
  coarse_loss_clip: true
  ema_momentum: 0.999
  optimizer: adam
  other_fix: false # it's needed for checking on multisong dataset if other is actually instrumental
  use_amp: true # enable or disable usage of mixed precision (float16) - usually it must be true

```

--------------------------------

### Train Music Source Separation Model

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/README.md

Example command to train a music source separation model. Specify the model type, configuration path, checkpoint, results path, data paths, and device IDs.

```bash
python train.py \
    --model_type mel_band_roformer \
    --config_path configs/config_mel_band_roformer_vocals.yaml \
    --start_check_point results/model.ckpt \
    --results_path results/ \
    --data_path 'datasets/dataset1' 'datasets/dataset2' \
    --valid_path datasets/musdb18hq/test \
    --num_workers 4 \
    --device_ids 0
```

--------------------------------

### Perform Inference for Music Source Separation

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/README.md

Example command to perform inference using a trained music source separation model. Specify the model type, configuration, checkpoint, input folder, and output directory.

```bash
python inference.py \
    --model_type mdx23c \
    --config_path configs/config_mdx23c_musdb18.yaml \
    --start_check_point results/last_mdx23c.ckpt \
    --input_folder input/wavs/ \
    --store_dir separation_results/
```

--------------------------------

### Training Invocation with CSV Dataset

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Example of how to invoke the training script using a CSV file for dataset definition. This specifies the model type, config path, data paths, dataset type, validation path, and device IDs.

```bash
python train.py \
    --model_type mel_band_roformer \
    --config_path configs/config_musdb18_mel_band_roformer.yaml \
    --results_path results/ \
    --data_path datasets/my_stems.csv datasets/extra_stems.csv \
    --dataset_type 3 \
    --valid_path datasets/musdb18hq/test \
    --device_ids 0

```

--------------------------------

### Run Basic Test Script

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/test.md

Execute the test script with essential arguments for model type, configuration, data paths, and checkpoint.

```bash
python tests.py --check_train \
--config_path config.yaml \
--model_type scnet \
--data_path /path/to/data \
--valid_path /path/to/valid 
```

--------------------------------

### Programmatic Ensemble with `ensemble_files`

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Execute audio file ensembling programmatically using the `ensemble_files` function. Arguments are passed as a list of strings.

```python
from ensemble import ensemble_files

ensemble_files([
    '--files', 'vocals_a.wav', 'vocals_b.wav',
    '--type', 'avg_wave',
    '--weights', '1', '1',
    '--output', 'result.wav'
])
```

--------------------------------

### Run Ensemble Script

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/ensemble.md

Use this command to ensemble audio files with specified weights and ensemble type. Ensure the number of weights matches the number of files.

```bash
ensemble.py --files ./results_tracks/vocals1.wav ./results_tracks/vocals2.wav --weights 2 1 --type max_fft --output out.wav
```

--------------------------------

### Single-GPU Training with `train.py`

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Entry point for training a model end-to-end on a single GPU. Handles data loading, optimization, AMP, EMA, and validation with checkpoint saving.

```bash
# Basic single-GPU training (Mel-Band RoFormer, vocals separation)
python train.py \
    --model_type mel_band_roformer \
    --config_path configs/config_musdb18_mel_band_roformer.yaml \
    --results_path results/ \
    --data_path datasets/musdb18hq/train \
    --valid_path datasets/musdb18hq/test \
    --num_workers 4 \
    --device_ids 0

# Resume from a checkpoint, monitor multiple metrics
python train.py \
    --model_type bs_roformer \
    --config_path configs/config_musdb18_bs_roformer.yaml \
    --start_check_point results/model_bs_roformer_ep_50_sdr_9.1234.ckpt \
    --load_optimizer --load_scheduler --load_epoch --load_best_metric \
    --results_path results/ \
    --data_path datasets/musdb18hq/train \
    --valid_path datasets/musdb18hq/test \
    --metrics sdr si_sdr l1_freq \
    --metric_for_scheduler sdr \
    --device_ids 0 1 \
    --num_workers 4

# Training with multiple data paths and dataset type 2 (stem folders)
python train.py \
    --model_type scnet \
    --config_path configs/config_musdb18_scnet.yaml \
    --results_path results/scnet/ \
    --data_path datasets/musdb18hq/train datasets/extra_stems \
    --dataset_type 2 \
    --valid_path datasets/musdb18hq/test \
    --device_ids 0 \
    --wandb_key YOUR_WANDB_KEY
# Checkpoints are saved to results/ as:
#   model_bs_roformer_ep_<N>_sdr_<value>.ckpt  (best only)
#   last_model_<type>.ckpt                      (always saved)
```

--------------------------------

### Training with LoRA Enabled

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/LoRA.md

Use the `--train_lora` flag when running the training script to enable LoRA fine-tuning.

```bash
python train.py --model_type scnet \
  --config_path configs/config_musdb18_scnet_large_starrytong.yaml \
  --start_check_point weights/last_scnet.ckpt \
  --results_path results/ \
  --data_path datasets/moisesdb/train_tracks \
  --valid_path datasets/moisesdb/valid \
  --device_ids 0 \
  --metrics neg_log_wmse l1_freq sdr \
  --metric_for_scheduler neg_log_wmse \
  --train_lora
```

--------------------------------

### Weights & Biases Experiment Tracking Initialization

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Python code demonstrating how to initialize Weights & Biases (wandb) for experiment tracking. Supports online, offline, and disabled modes based on `wandb_key` and `wandb_offline` arguments. Requires `argparse` and utilities from `utils.settings`.

```python
import argparse
from utils.settings import wandb_init, get_model_from_config

_, config = get_model_from_config('bs_roformer', 'configs/config_musdb18_bs_roformer.yaml')

args = argparse.Namespace(
    model_type='bs_roformer',
    device_ids=[0],
    wandb_key='YOUR_WANDB_API_KEY',  # set '' or None to disable
    wandb_offline=False,
)

# Online mode (requires valid API key)
wandb_init(args, config, batch_size=10)

# Offline mode (no network required, sync later with `wandb sync`)
args.wandb_offline = True
wandb_init(args, config, batch_size=10)

# Disabled (no tracking at all)
args.wandb_key = ''
args.wandb_offline = False
wandb_init(args, config, batch_size=10)
# Run name is auto-generated: "bs_roformer_[vocals-bass-drums-other]_2024-06-01"
```

--------------------------------

### Multi-Metric Validation with Spectrograms

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Perform validation with multiple metrics, save results to a directory, and generate spectrograms. Supports multi-GPU validation.

```bash
python valid.py \
    --model_type mel_band_roformer \
    --config_path configs/config_musdb18_mel_band_roformer.yaml \
    --start_check_point weights/model_mel_band_roformer.ckpt \
    --valid_path datasets/musdb18hq/test \
    --store_dir results_valid/ \
    --draw_spectro 30 \
    --metrics sdr si_sdr l1_freq aura_stft bleedless fullness \
    --device_ids 0 1
```

--------------------------------

### Basic Inference with `inference.py`

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Perform basic audio separation on files in a folder. Specify model type, configuration, checkpoint, input, and output directories.

```bash
python inference.py \
    --model_type mel_band_roformer \
    --config_path configs/config_musdb18_mel_band_roformer.yaml \
    --start_check_point weights/model_mel_band_roformer_ep_300_sdr_11.ckpt \
    --input_folder input/songs/ \
    --store_dir output/separated/
```

--------------------------------

### Load Model from Config using `get_model_from_config`

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Use this function to load and instantiate a PyTorch model from a YAML configuration file. It's the central entry point for training, validation, and inference scripts.

```python
from utils.settings import get_model_from_config

# Load a Band-Split RoFormer model
model, config = get_model_from_config(
    model_type='bs_roformer',
    config_path='configs/config_musdb18_bs_roformer.yaml'
)
print(model)
# BSRoformer(dim=192, depth=6, stereo=True, ...)

# Load a Mel-Band RoFormer model
model, config = get_model_from_config(
    model_type='mel_band_roformer',
    config_path='configs/config_musdb18_mel_band_roformer.yaml'
)

# Load a Demucs4HT model
model, config = get_model_from_config(
    model_type='htdemucs',
    config_path='configs/config_musdb18_htdemucs.yaml'
)

# Supported model_type values:
# 'mdx23c', 'htdemucs', 'segm_models', 'torchseg',
# 'mel_band_roformer', 'mel_band_conformer', 'bs_roformer', 'bs_conformer',
# 'bs_mamba2', 'swin_upernet', 'bandit', 'bandit_v2',
# 'scnet', 'scnet_tran', 'scnet_unofficial', 'scnet_masked',
# 'apollo', 'conformer', 'moises_light'
```

--------------------------------

### Train BSRoformer with LoRA

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/LoRA.md

Use this command to finetune BSRoformer on your dataset with LoRA enabled. Ensure you have the specified config file and pre-trained weights.

```bash
python train.py --model_type bs_roformer \
  --config_path configs/config_musdb18_bs_roformer_with_lora.yaml \
  --start_check_point weights/model_bs_roformer_ep_17_sdr_9.6568.ckpt \
  --results_path results/ \
  --data_path musdb18hq/train \
  --valid_path musdb18hq/test \
  --device_ids 0 \
  --metrics sdr \
  --train_lora
```

--------------------------------

### Programmatic DDP Launch

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Launch distributed data parallel training programmatically. Ensure the 'train_ddp' module is available and configured correctly.

```python
from train_ddp import train_model_ddp

train_model_ddp({
    'model_type': 'bs_roformer',
    'config_path': 'configs/config_musdb18_bs_roformer.yaml',
    'results_path': 'results/',
    'data_path': ['datasets/musdb18hq/train'],
    'valid_path': ['datasets/musdb18hq/test'],
    'num_workers': 4,
    'metrics': ['sdr'],
    'metric_for_scheduler': 'sdr',
})
```

--------------------------------

### LoRA Configuration Parameters

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/LoRA.md

Configure LoRA settings in your config file. Adjust 'r' for rank, 'lora_alpha' for scaling, and 'lora_dropout' for regularization. 'merge_weights' and 'fan_in_fan_out' control weight merging and initialization. 'enable_lora' specifies which layers LoRA is applied to.

```yaml
lora:
  r: 8
  lora_alpha: 16 # alpha / rank > 1
  lora_dropout: 0.05
  merge_weights: False
  fan_in_fan_out: False
  enable_lora: [True]
```

--------------------------------

### Fine-tune BS-RoFormer with LoRA using loralib

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Command to fine-tune a BS-RoFormer model with LoRA on a custom dataset. Ensure the config path and dataset paths are correctly specified. The `--train_lora_loralib` flag activates LoRA training via the loralib backend.

```bash
python train.py \
    --model_type bs_roformer \
    --config_path configs/config_musdb18_bs_roformer_with_lora.yaml \
    --start_check_point weights/model_bs_roformer_ep_17_sdr_9.6568.ckpt \
    --results_path results/lora_finetune/ \
    --data_path datasets/my_custom_vocals/train \
    --valid_path datasets/my_custom_vocals/test \
    --device_ids 0 \
    --metrics sdr \
    --train_lora_loralib
```

--------------------------------

### Audio Demixing and BigShifts Wrapper

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Python code for loading a model and performing audio source separation using `demix` for standard chunk processing and `bigshifts_wrapper` for enhanced quality through averaged time-shifted passes. Requires `torch` and `librosa`.

```python
import torch
import librosa
from utils.settings import get_model_from_config
from utils.model_utils import demix, bigshifts_wrapper

# Load model
model, config = get_model_from_config('bs_roformer', 'configs/config_musdb18_bs_roformer.yaml')
checkpoint = torch.load('weights/model_bs_roformer.ckpt', map_location='cpu')
model.load_state_dict(checkpoint['state_dict'])
model.eval()

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
model.to(device)

# Load audio (stereo, 44100 Hz)
mix, sr = librosa.load('song.wav', sr=44100, mono=False)  # shape: (2, N)

# Standard demix
sources = demix(config, model, mix, device, model_type='bs_roformer', pbar=True)
# sources = {'vocals': array(2, N), 'bass': array(2, N), 'drums': array(2, N), 'other': array(2, N)}

# BigShifts demix (averages 3 time-shifted passes for better quality)
sources = bigshifts_wrapper(config, model, mix, device, model_type='bs_roformer', bigshifts=3, pbar=True)

import soundfile as sf
sf.write('vocals.flac', sources['vocals'].T, sr, subtype='PCM_16')
```

--------------------------------

### Ensemble Separation Outputs (Weighted FFT Max)

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Perform a weighted maximum ensemble in the FFT domain for aggressive separation. Requires multiple source files and corresponding weights.

```bash
python ensemble.py \
    --files vocals_model1.wav vocals_model2.wav vocals_model3.wav \
    --weights 2 1 1 \
    --type max_fft \
    --output best_vocals.wav
```

--------------------------------

### Full Augmentations Configuration

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/augmentations.md

This configuration enables and defines parameters for a comprehensive set of audio augmentations. It controls loudness, mixup, MP3 compression, chunk size, and various effects like reverb, chorus, distortion, pitch shifting, and more, applied globally or to specific instrument types.

```yaml
augmentations:
  enable: true
  loudness: true
  loudness_min: 0.5
  loudness_max: 1.5
  mixup: true
  mixup_probs: !!python/tuple
    - 0.2
    - 0.02
  mixup_loudness_min: 0.5
  mixup_loudness_max: 1.5
  mp3_compression_on_mixture: 0.01
  mp3_compression_on_mixture_bitrate_min: 32
  mp3_compression_on_mixture_bitrate_max: 320
  mp3_compression_on_mixture_backend: "lameenc"
  chunk_size_augm: true
  chunk_size_min: 44100
  chunk_size_max: 661500
  all:
    channel_shuffle: 0.5
    random_inverse: 0.1
    random_polarity: 0.5
    mp3_compression: 0.01
    mp3_compression_min_bitrate: 32
    mp3_compression_max_bitrate: 320
    mp3_compression_backend: "lameenc"
    pedalboard_reverb: 0.01
    pedalboard_reverb_room_size_min: 0.1
    pedalboard_reverb_room_size_max: 0.9
    pedalboard_reverb_damping_min: 0.1
    pedalboard_reverb_damping_max: 0.9
    pedalboard_reverb_wet_level_min: 0.1
    pedalboard_reverb_wet_level_max: 0.9
    pedalboard_reverb_dry_level_min: 0.1
    pedalboard_reverb_dry_level_max: 0.9
    pedalboard_reverb_width_min: 0.9
    pedalboard_reverb_width_max: 1.0
    pedalboard_chorus: 0.01
    pedalboard_chorus_rate_hz_min: 1.0
    pedalboard_chorus_rate_hz_max: 7.0
    pedalboard_chorus_depth_min: 0.25
    pedalboard_chorus_depth_max: 0.95
    pedalboard_chorus_centre_delay_ms_min: 3
    pedalboard_chorus_centre_delay_ms_max: 10
    pedalboard_chorus_feedback_min: 0.0
    pedalboard_chorus_feedback_max: 0.5
    pedalboard_chorus_mix_min: 0.1
    pedalboard_chorus_mix_max: 0.9
    pedalboard_phazer: 0.01
    pedalboard_phazer_rate_hz_min: 1.0
    pedalboard_phazer_rate_hz_max: 10.0
    pedalboard_phazer_depth_min: 0.25
    pedalboard_phazer_depth_max: 0.95
    pedalboard_phazer_centre_frequency_hz_min: 200
    pedalboard_phazer_centre_frequency_hz_max: 12000
    pedalboard_phazer_feedback_min: 0.0
    pedalboard_phazer_feedback_max: 0.5
    pedalboard_phazer_mix_min: 0.1
    pedalboard_phazer_mix_max: 0.9
    pedalboard_distortion: 0.01
    pedalboard_distortion_drive_db_min: 1.0
    pedalboard_distortion_drive_db_max: 25.0
    pedalboard_pitch_shift: 0.01
    pedalboard_pitch_shift_semitones_min: -7
    pedalboard_pitch_shift_semitones_max: 7
    pedalboard_resample: 0.01
    pedalboard_resample_target_sample_rate_min: 4000
    pedalboard_resample_target_sample_rate_max: 44100
    pedalboard_bitcrash: 0.01
    pedalboard_bitcrash_bit_depth_min: 4
    pedalboard_bitcrash_bit_depth_max: 16
    pedalboard_mp3_compressor: 0.01
    pedalboard_mp3_compressor_pedalboard_mp3_compressor_min: 0
    pedalboard_mp3_compressor_pedalboard_mp3_compressor_max: 9.999
  vocals:
    pitch_shift: 0.1
    pitch_shift_min_semitones: -5
    pitch_shift_max_semitones: 5
    seven_band_parametric_eq: 0.25
    seven_band_parametric_eq_min_gain_db: -9
    seven_band_parametric_eq_max_gain_db: 9
    tanh_distortion: 0.1
    tanh_distortion_min: 0.1
    tanh_distortion_max: 0.7
  bass:
    pitch_shift: 0.1
    pitch_shift_min_semitones: -2
    pitch_shift_max_semitones: 2
    seven_band_parametric_eq: 0.25
    seven_band_parametric_eq_min_gain_db: -3
    seven_band_parametric_eq_max_gain_db: 6
    tanh_distortion: 0.2
    tanh_distortion_min: 0.1
    tanh_distortion_max: 0.5
  drums:
    pitch_shift: 0.33
    pitch_shift_min_semitones: -5
    pitch_shift_max_semitones: 5
    seven_band_parametric_eq: 0.25
    seven_band_parametric_eq_min_gain_db: -9
    seven_band_parametric_eq_max_gain_db: 9
    tanh_distortion: 0.33
    tanh_distortion_min: 0.1
    tanh_distortion_max: 0.6
  other:
    pitch_shift: 0.1
    pitch_shift_min_semitones: -4
    pitch_shift_max_semitones: 4
    gaussian_noise: 0.1
    gaussian_noise_min_amplitude: 0.001
    gaussian_noise_max_amplitude: 0.015
    time_stretch: 0.01
    time_stretch_min_rate: 0.8
    time_stretch_max_rate: 1.25
```

--------------------------------

### Inference with LoRA Checkpoint

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/LoRA.md

For inference with a LoRA-tuned model, provide the path to the LoRA checkpoint via the `--lora_checkpoint` argument.

```bash
python inference.py --lora_checkpoint weights/lora_last_scnet.ckpt \
  --model_type scnet \
  --config_path configs/config_musdb18_scnet_large_starrytong.yaml \
  --start_check_point weights/last_scnet.ckpt \
  --store_dir inference_results/ \
  --input_folder datasets/moisesdb/mixtures_for_inference \
  --device_ids 0
```

--------------------------------

### Run Comprehensive Test Script

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/test.md

Perform a full validation including training, validation, and inference checks, with options for test-time augmentation and specific loss functions.

```bash
python tests/test.py \
--check_train \
--check_valid \
--check_inference \
--model_type scnet \
--config_path configs/config_musdb18_scnet_large_starrytong.yaml \
--start_check_point weights/model_scnet_ep_30_neg_log_wmse_-11.8688.ckpt \
--data_path datasets/moisesdb/train_tracks \
--valid_path datasets/moisesdb/valid \
--use_tta \
--use_mse_loss
```

--------------------------------

### Multi-GPU Distributed Data Parallel Training with `train_ddp.py`

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Launches training processes across multiple GPUs using Distributed Data Parallel (DDP) for significantly faster training. Wraps the single-GPU training logic.

```bash
# Launch DDP training across all available GPUs automatically
python train_ddp.py \
    --model_type mel_band_roformer \
    --config_path configs/config_musdb18_mel_band_roformer.yaml \
    --results_path results/ \
    --data_path datasets/musdb18hq/train \
    --valid_path datasets/musdb18hq/test \
    --num_workers 4
```

--------------------------------

### Programmatic Inference with `proc_folder`

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Execute audio separation programmatically using the `proc_folder` function. Configure model, paths, and processing options.

```python
from inference import proc_folder

proc_folder({
    'model_type': 'bs_roformer',
    'config_path': 'configs/config_musdb18_bs_roformer.yaml',
    'start_check_point': 'weights/model_bs_roformer.ckpt',
    'input_folder': 'input/songs/',
    'store_dir': 'output/separated/',
    'use_tta': True,
    'device_ids': [0],
    'bigshifts': 1,
})
```

--------------------------------

### Advanced Inference with TTA and Instrumental Extraction

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Run inference with test-time augmentation, instrumental extraction, and custom output naming. Supports BigShifts averaging and device selection.

```bash
python inference.py \
    --model_type bs_roformer \
    --config_path configs/config_musdb18_bs_roformer.yaml \
    --start_check_point weights/model_bs_roformer.ckpt \
    --input_folder input/songs/ \
    --store_dir output/ \
    --use_tta \
    --extract_instrumental \
    --bigshifts 3 \
    --filename_template "{file_name}/{model}/{instr}" \
    --device_ids 0
```

--------------------------------

### Stems Dataset Structure

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/dataset_types.md

Organizes audio files by stem type, with each folder containing only files for a specific instrument.

```text
--- vocals:
------ vocals_1.wav
------ vocals_2.wav
------ vocals_3.wav
------ vocals_4.wav
------ ...
--- bass:
------ bass_1.wav
------ bass_2.wav
------ bass_3.wav
------ bass_4.wav
------ ...
...........
```

--------------------------------

### Inference with a LoRA checkpoint

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Command to perform inference using a base model combined with LoRA weights. Provide the base model config, base checkpoint, and the LoRA checkpoint. The input and output directories should also be specified.

```bash
python inference.py \
    --model_type scnet \
    --config_path configs/config_musdb18_scnet.yaml \
    --start_check_point weights/base_scnet.ckpt \
    --lora_checkpoint_loralib weights/lora_scnet.ckpt \
    --input_folder input/songs/ \
    --store_dir output/lora_results/
```

--------------------------------

### CSV File Dataset Structure

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/dataset_types.md

Uses a CSV file to map instrument names to the paths of their corresponding audio files.

```text
instrum,path
vocals,/path/to/dataset/vocals_1.wav
vocals,/path/to/dataset2/vocals_v2.wav
vocals,/path/to/dataset3/vocals_some.wav
...
drums,/path/to/dataset/drums_good.wav
...
```

--------------------------------

### Model Configuration YAML

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Defines model architecture and training hyperparameters. This config is loaded with `get_model_from_config` and used throughout training, validation, and inference.

```yaml
# configs/config_musdb18_bs_roformer.yaml — Band-Split RoFormer example
audio:
  chunk_size: 131584        # Training chunk length in samples
  dim_f: 1024               # Frequency dimension for STFT
  hop_length: 512
  n_fft: 2048
  num_channels: 2           # Stereo
  sample_rate: 44100
  min_mean_abs: 0.001       # Skip silent chunks below this threshold

model:
  dim: 192
  depth: 6
  stereo: true
  num_stems: 1
  time_transformer_depth: 1
  freq_transformer_depth: 1
  heads: 8
  dim_head: 64
  attn_dropout: 0.1
  ff_dropout: 0.1
  flash_attn: true          # Use Flash Attention for memory efficiency
  stft_n_fft: 2048
  stft_hop_length: 512
  mask_estimator_depth: 2
  multi_stft_resolution_loss_weight: 1.0

training:
  batch_size: 10
  gradient_accumulation_steps: 1
  grad_clip: 0
  instruments: [vocals, bass, drums, other]
  lr: 5.0e-05
  patience: 2               # LR reduce patience (ReduceLROnPlateau)
  reduce_factor: 0.95
  target_instrument: vocals # Primary separation target
  num_epochs: 1000
  num_steps: 1000
  ema_momentum: 0.999       # Exponential moving average decay
  optimizer: adam
  use_amp: true             # Mixed precision (float16) 

augmentations:
  enable: true
  loudness: true
  loudness_min: 0.5
  loudness_max: 1.5

inference:
  batch_size: 1
  num_overlap: 4            # Overlap factor for chunked inference

```

--------------------------------

### MUSDB Dataset Structure

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/dataset_types.md

Represents a dataset where each song is in its own folder, containing all necessary stems as individual audio files.

```text
--- Song 1:
------ vocals.wav
------ bass.wav
------ drums.wav
------ other.wav
--- Song 2:
------ vocals.wav
------ bass.wav
------ drums.wav
------ other.wav
--- Song 3:
...........
```

--------------------------------

### Ensemble Separation Outputs (Waveform Average)

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Combine multiple separation results using waveform averaging. Specify input files and output file. Equal weights are used by default.

```bash
python ensemble.py \
    --files results_model1/vocals.wav results_model2/vocals.wav \
    --type avg_wave \
    --output ensemble_vocals.wav
```

--------------------------------

### MUSDB Aligned Dataset Structure

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/dataset_types.md

Similar to MUSDB, but all stems are loaded from the same song position during training.

```text
--- Song 1:
------ vocals.wav
------ bass.wav
------ drums.wav
------ other.wav
```

--------------------------------

### LoRA Configuration for PEFT

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Add this YAML block to your configuration file to enable LoRA fine-tuning. Adjust rank, scaling factor, dropout, and specify which sub-projections to apply LoRA to.

```yaml
lora:
  r: 8                   # Rank of the low-rank matrices (smaller = fewer params)
  lora_alpha: 16         # Scaling factor; lora_alpha/r should be > 1
  lora_dropout: 0.05     # Regularization dropout on LoRA layers
  merge_weights: false   # Set true to bake LoRA into base weights for deployment
  fan_in_fan_out: false
  enable_lora: [true]    # Which sub-projections get LoRA (e.g. [true, false, true] for Q/V only)
```

--------------------------------

### Augmentations Config YAML

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Defines on-the-fly data augmentation techniques. Augmentations can be applied to all stems or specific instrument stems.

```yaml
# Full augmentation example (paste into any config)
augmentations:
  enable: true
  loudness: true
  loudness_min: 0.5
  loudness_max: 1.5
  mixup: true
  mixup_probs: !!python/tuple
    - 0.2    # Probability of mixing in 1 extra same-type stem
    - 0.02   # Probability of mixing in 2 extra same-type stems
  mixup_loudness_min: 0.5
  mixup_loudness_max: 1.5

  # Simulate downloading mp3 from the internet
  mp3_compression_on_mixture: 0.01
  mp3_compression_on_mixture_bitrate_min: 32
  mp3_compression_on_mixture_bitrate_max: 320
  mp3_compression_on_mixture_backend: "lameenc"

  # Random chunk sizes (careful with VRAM)
  chunk_size_augm: true
  chunk_size_min: 44100
  chunk_size_max: 661500

  all:                         # Applied to every stem
    channel_shuffle: 0.5
    random_polarity: 0.5
    mp3_compression: 0.01
    mp3_compression_min_bitrate: 32
    mp3_compression_max_bitrate: 320
    pedalboard_reverb: 0.01
    pedalboard_chorus: 0.01
    pedalboard_pitch_shift: 0.01
    pedalboard_pitch_shift_semitones_min: -7
    pedalboard_pitch_shift_semitones_max: 7

  vocals:                      # Applied only to the vocals stem
    pitch_shift: 0.1
    pitch_shift_min_semitones: -5
    pitch_shift_max_semitones: 5
    seven_band_parametric_eq: 0.25
    seven_band_parametric_eq_min_gain_db: -9
    seven_band_parametric_eq_max_gain_db: 9
    tanh_distortion: 0.1
    tanh_distortion_min: 0.1
    tanh_distortion_max: 0.7

  drums:
    pitch_shift: 0.33
    pitch_shift_min_semitones: -5
    pitch_shift_max_semitones: 5

```

--------------------------------

### MUSDB Aligned + Explicit Mixture Dataset Structure

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/dataset_types.md

Extends MUSDB Aligned by explicitly including a `mixture.wav` file, treated as a separate signal.

```text
--- Song 1:
------ vocals.wav
------ bass.wav
------ drums.wav
------ other.wav
------ mixture.wav
```

--------------------------------

### Validating with LoRA Checkpoint

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/LoRA.md

When validating a model fine-tuned with LoRA, specify the LoRA checkpoint using the `--lora_checkpoint` parameter.

```bash
python valid.py --model_type scnet \
  --config_path configs/config_musdb18_scnet_large_starrytong.yaml \
  --start_check_point weights/last_scnet.ckpt \
  --store_dir results_store/ \
  --valid_path datasets/moisesdb/valid \
  --device_ids 0 \
  --metrics neg_log_wmse l1_freq si_sdr sdr aura_stft aura_mrstft bleedless fullness
```

--------------------------------

### Validate Model with SDR Metric

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Evaluate a trained model against a validation dataset using the SDR metric. Specify model, config, checkpoint, and validation path.

```bash
python valid.py \
    --model_type bs_roformer \
    --config_path configs/config_musdb18_bs_roformer.yaml \
    --start_check_point weights/model_bs_roformer_ep_317_sdr_12.9755.ckpt \
    --valid_path datasets/musdb18hq/test \
    --device_ids 0
```

--------------------------------

### Programmatic Validation with `check_validation`

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Execute model validation programmatically using the `check_validation` function. Returns average and detailed metrics.

```python
from valid import check_validation

metrics_avg, all_metrics = check_validation({
    'model_type': 'mel_band_roformer',
    'config_path': 'configs/config_musdb18_mel_band_roformer.yaml',
    'start_check_point': 'weights/model.ckpt',
    'valid_path': ['datasets/musdb18hq/test'],
    'metrics': ['sdr', 'si_sdr'],
    'device_ids': [0],
})
```

--------------------------------

### Validate a LoRA-fine-tuned model

Source: https://context7.com/zfturbo/music-source-separation-training/llms.txt

Command to validate a model that has been fine-tuned with LoRA. Specify the base model config, the base model checkpoint, and the LoRA checkpoint. The `--lora_checkpoint_loralib` argument is crucial for loading the LoRA weights.

```bash
python valid.py \
    --model_type scnet \
    --config_path configs/config_musdb18_scnet.yaml \
    --start_check_point weights/base_scnet.ckpt \
    --lora_checkpoint_loralib weights/lora_scnet.ckpt \
    --valid_path datasets/musdb18hq/test \
    --metrics sdr si_sdr l1_freq
```

--------------------------------

### Validation Dataset Structure

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/dataset_types.md

The validation dataset must follow the Type 1 structure and include a `mixture.wav` for each song.

```text
--- Song 1:
------ vocals.wav  
------ bass.wav 
------ drums.wav
------ other.wav
------ mixture.wav
--- Song 2:
------ vocals.wav  
------ bass.wav 
------ drums.wav
------ other.wav
------ mixture.wav
--- Song 3:
...........
```

--------------------------------

### Class-Balanced Aligned Dataset Structure

Source: https://github.com/zfturbo/music-source-separation-training/blob/main/docs/dataset_types.md

A class-balanced dataset structure, similar to Type 6, designed to mitigate class frequency bias.

```text
--- Song 1:
------ flute.wav
------ violin.wav
------ mixture.wav
```
Tokens:
Raw