TuneJury

https://github.com/yonghyunk1m/tunejury
Admin
TuneJury is an open pairwise reward model for text-to-music generation designed to evaluate music...
Tokens:44,017
Snippets:237
Trust Score:6.7
Update:1 week ago
Show doc for...
Tokens:
Raw
### Quick start: clone, install, and score audio with TuneJury

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/project_page.html

Clones the TuneJury repository, installs dependencies, and runs a Python snippet that loads the scorer and scores an audio file with an optional prompt. An empty prompt is OOD-safe as described in paper section 4.2.

```bash
git clone https://github.com/yonghyunk1m/TuneJury.git
cd TuneJury
pip install -r requirements.txt

```

```python
python -c "
from tunejury import Scorer
scorer = Scorer.from_pretrained('checkpoints/tunejury.pt')
print(scorer.score('your_audio.wav', prompt='your caption'))
print(scorer.score('your_audio.wav', prompt=''))   # empty prompt -> OOD-safe (paper 4.2)
"

```

--------------------------------

### Install system dependencies (Debian/Ubuntu)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Installs ffmpeg, libsndfile1, and git-lfs. ffmpeg is required by torchaudio and all Mode 1/2/3 generation scripts; libsndfile1 is required by soundfile. git-lfs is only needed for cloning checkpoint blobs from the GitHub mirror; the canonical download from Hugging Face Hub does not require it.

```bash
sudo apt-get install -y ffmpeg libsndfile1 git-lfs
git lfs install
```

--------------------------------

### Run anchor calibration experiments (K-sweep, retrain, cross-month, figure, diagnostics, forgetting probe)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Commands to reproduce all anchor calibration experiments. Requires completed setup, committed data splits/labels, and pre-extracted CLAP/MERT features. Replace /path/to/musicarena/features and /path/to/april/{features,json} with local paths. Expected results are listed below the commands.

```bash
# Headline anchor (A) row: K-sweep on the 598-pair Feb-Mar post-cutoff slice,
# 50/50 held-out split, 5 seeds, K ∈ {0, 3, 10, 30, 100, 250}
python -m applications.anchor_calibration.run_experiment \
    --checkpoint checkpoints/tunejury.pt \
    --feat-dir /path/to/musicarena/features \
    --increment-split data/splits/MusicArena_v2_increment.json \
    --label-csv data/labels/ma_postcut_scored.csv \
    --out results/ood_repro.json

# Retrain (R) row: from-scratch head training on bench_clean ∪ K added pairs,
# same 50/50 split and seeds as the anchor row
python -m applications.anchor_calibration.retrain_ksweep \
    --feat-dir /path/to/musicarena/features \
    --bench-split data/splits/MusicArena-random_split_bench_clean.json \
    --increment-split data/splits/MusicArena_v2_increment.json \
    --label-csv data/labels/ma_postcut_scored.csv \
    --out results/ood_retrain.json

# Cross-month transfer: Feb-Mar β_s applied to April 2026 (Table cross_month)
python -m applications.anchor_calibration.cross_month_eval \
    --checkpoint checkpoints/tunejury.pt \
    --feat-dir /path/to/musicarena/features \
    --label-csv data/labels/ma_postcut_scored.csv \
    --april-feat-dir /path/to/april/features \
    --april-meta-dir /path/to/april/json \
    --out results/april_cross_month.json

# Render Figure `fig:ood_scaling` (Figure 5)
python figures/make_ood_scaling_figure.py \
    --anchor-json     results/ood_repro.json \
    --retrain-json    results/ood_retrain.json \
    --cross-month-json results/april_cross_month.json \
    --out             figures/ood_scaling.pdf

# Diagnostics + behavior probes (text-level claims in Appendix D, not in tab:ood_scaling)
python -m applications.anchor_calibration.intrinsic_difficulty \
    --checkpoint     checkpoints/tunejury.pt \
    --feat-dir-fm    /path/to/musicarena/features_feb_mar \
    --feat-dir-apr   /path/to/musicarena/features_april \
    --bench-test     data/splits/MusicArena-random_split_bench_clean.json \
    --increment-uuids data/splits/MusicArena_v2_increment.json
python -m applications.anchor_calibration.postcut_diagnostics \
    --checkpoint     checkpoints/tunejury.pt \
    --feat-dir       /path/to/musicarena/features \
    --training-split data/splits/MusicArena-random_split_bench_clean.json \
    --label-csv      data/labels/ma_postcut_scored.csv
python -m applications.anchor_calibration.heldout_margin \
    --checkpoint   checkpoints/tunejury.pt \
    --features_dir data/processed_features \
    --test_ids     data/splits/test.txt

# "No catastrophic forgetting" probe (§A.D): naive-retraining baseline
# trained from scratch on bench_clean (571) ∪ Feb-Mar (598) ∪ April (397),
# evaluated on three held-out partitions (3 seeds). Backs the "+5 pp
# Feb-Mar, +7 pp April, bench-clean within noise" claim in the paper.
python -m applications.anchor_calibration.retrain_forgetting \
    --feat-dir-fm     /path/to/musicarena/features_feb_mar \
    --feat-dir-apr    /path/to/musicarena/features_april \
    --bench-split     data/splits/MusicArena-random_split_bench_clean.json \
    --increment-split data/splits/MusicArena_v2_increment.json \
    --fm-label-csv    data/labels/ma_postcut_scored.csv
```

--------------------------------

### Recompute MAD for SAO-small

Source: https://github.com/yonghyunk1m/tunejury/blob/main/applications/mode2_ditto/README.md

Splits the flat output into baseline and ditto directories, then runs mad_compute.py. Requires pip install mauve-text. MAD is -ln(MAUVE) against the SDD-706 reference set.

```bash
mkdir -p results/ditto_sao_small/baseline results/ditto_sao_small/ditto
mv results/ditto_sao_small/*_baseline.wav results/ditto_sao_small/baseline/
mv results/ditto_sao_small/*_ditto.wav    results/ditto_sao_small/ditto/

python ../mode3_expert_iter/mad_compute.py \
    --baseline_dir results/ditto_sao_small/baseline \
    --after_ft_dir results/ditto_sao_small/ditto \
    --sdd706_dir   /path/to/SDD-706/audio_wav_torch \
    --out_json     mad_sao_small.json
```

--------------------------------

### Install TuneJury with conda and Hugging Face login

Source: https://github.com/yonghyunk1m/tunejury/blob/main/README.md

Clones the repository, creates and activates the 'tunejury' conda environment, and logs in to the Hugging Face CLI because MERT-v1-330M is a gated model.

```bash
git clone https://github.com/yonghyunk1m/TuneJury
cd TuneJury
conda env create -f environment.yml   # creates env "tunejury"
conda activate tunejury
huggingface-cli login                 # MERT-v1-330M is gated
```

--------------------------------

### Run MAD computation with mad_compute.py

Source: https://github.com/yonghyunk1m/tunejury/blob/main/applications/mode3_expert_iter/README.md

Computes MAD on MERT-v1-330M embeddings against the SDD-706 reference set for both the pretrained baseline and the expert-iteration post-trained checkpoint. Requires pip install mauve-text. Each --*_dir should contain ~100 wav files (one per SDD-100 prompt, generated with CFG 4.5 + 25 Euler steps, no post-processing) consistent with the inference settings used for Table 5.

```bash
python mad_compute.py \
    --baseline_dir /path/to/pretrained_FluxAudio_S/sdd100_outputs \
    --after_ft_dir /path/to/expert_iter_FluxAudio_S/sdd100_outputs \
    --sdd706_dir   /path/to/SDD-706/audio_wav_torch \
    --out_json     mad_mode3_summary.json
```

--------------------------------

### End-to-end best-of-N on MusicGen-medium (generate, score, aggregate)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/applications/mode1_bon/README.md

Three-step workflow: generate N=32 candidates per prompt with generate_musicgen.py, score with TuneJury and select top-1 at N ∈ {1,2,4,8,16,32} using score_and_select.py, then aggregate metrics with aggregate.py. The results.csv matches Figure 3 and Appendix Table tab:apps_mode1_bon_full.

```bash
# Step 1: generate N=32 candidates per prompt
python generate_musicgen.py \
    --prompts ../../eval/prompts/sdd100.json \
    --model facebook/musicgen-medium \
    --out_dir results/musicgen_medium_bon100 \
    --n_candidates 32 \
    --seeds 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73

# Step 2: score with TuneJury and select top-1 at N ∈ {1, 2, 4, 8, 16, 32}
python score_and_select.py \
    --prompts ../../eval/prompts/sdd100.json \
    --candidates_dir results/musicgen_medium_bon100 \
    --ckpt ../../checkpoints/tunejury.pt \
    --n_candidates 32 --N 1 2 4 8 16 32 \
    --per_cand_csv results/musicgen_medium_bon100/tunejury_per_candidate.csv \
    --out_csv results/musicgen_medium_bon100/results.csv

# Step 3: aggregate metrics into a single table
python aggregate.py results/musicgen_medium_bon100/results.csv
```

--------------------------------

### Load TuneJury release-scores dataset with datasets library

Source: https://github.com/yonghyunk1m/tunejury/blob/main/release_scores/README.md

Loads the 'mtg_jamendo' config of the TuneJury/release-scores dataset. Each config corresponds to a collection; the first training example contains track_id, reward_score, and other fields.

```python
from datasets import load_dataset

ds = load_dataset("TuneJury/release-scores", "mtg_jamendo")   # one config per collection
print(ds["train"][0])   # {'track_id': ..., 'reward_score': ..., ...}
```

--------------------------------

### End-to-end regeneration pipeline (bash)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Complete pipeline from prerequisites to building splits. Requires LAION-CLAP-Music checkpoint and ~6 GB free GPU memory. Steps include downloading audio, optional SongEval pair synthesis, optional bench-clean filtering, merging pair files, feature extraction, validation, and building splits.

```bash
# Step 0: prerequisites. LAION-CLAP-Music checkpoint, ~6 GB free GPU memory
ls checkpoints/music_audioset_epoch_15_esc_90.14.pt   # must exist (see §1)

# Step 1: download raw audio for all four sources (URLs above).
#   Suggested layout (matches downstream scripts):
#     /path/to/audio/MusicArena/<battle_uuid>_a.wav
#     /path/to/audio/MusicArena/<battle_uuid>_b.wav
#     /path/to/audio/MusicPrefs/<pair_id>_a.wav, _b.wav
#     /path/to/audio/AIME/<pair_id>_a.wav, _b.wav
#     /path/to/audio/SongEval/<song_id>_a.wav, _b.wav

# Step 2 (SongEval only): synthesize pairwise labels
python data/prepare_songeval_pairs.py \
    --songeval-root /path/to/songeval \
    --out-pairs data/splits/songeval_pairs.json \
    --mean-gap 0.5

# Step 3 (optional, for item-level disjointness with CMI-RewardBench):
python data/filter_music_arena_bench_clean.py \
    --music-arena-root /path/to/music_arena \
    --cmi-bench-uuid-list /path/to/cmi_rewardbench/music_arena_uuids.txt \
    --out-pool data/music_arena_bench_clean.json

# Step 4: merge the four per-source pair files into one all_pairs.json
#   (each record carries pair_id, source, winner, prompt, a_audio, b_audio).
#   This step is dataset-specific: see each source's README for field names.

# Step 5: feature extraction
python data/extract_features.py \
    --pairs data/splits/all_pairs.json \
    --audio-root /path/to/audio \
    --out-dir data/processed_features \
    --encoders clap mert

# Step 6: validate. The .pt count must be non-zero (and typically ~21.8 K)
python - <<'PY'
from pathlib import Path
n = sum(1 for _ in Path("data/processed_features").glob("*.pt"))
print(f"Found {n} .pt blobs in data/processed_features/")
assert n > 0, "extraction produced 0 files; see docs/reproducing.md §2 and data/processed_features/README.md"
PY

# Step 7: (re)build the 17 derived .txt splits consumed by tunejury.train
python -m data.build_splits
```

--------------------------------

### Compute MAD (MAUVE Audio Divergence) with mad_compute.py

Source: https://github.com/yonghyunk1m/tunejury/blob/main/applications/mode1_bon/README.md

Computes MAD on MERT-v1-330M embeddings against the SDD-706 reference set. Requires pip install mauve-text. Selects N=1 (first seed) and N=16 (top-1 by TuneJury) audio, then runs mauve.compute_mauve between each backbone's distribution and the SDD-706 reference.

```bash
python mad_compute.py \
    --bon_results_root /path/to/results \
    --sdd706_dir       /path/to/SDD-706/audio_wav_torch \
    --prompts          ../../eval/prompts/sdd100.json \
    --backbones        musicgen_medium musicgen_large audioldm2_music acestep \
    --out_json         mad_mode1_summary.json
```

--------------------------------

### Run CMI-RewardBench head-to-head evaluation

Source: https://github.com/yonghyunk1m/tunejury/blob/main/README.md

Reproduce the head-to-head results from section 4.2 / Table head_to_head against PAM score, Audiobox-Aesthetics, SongEval-RM, CMI-RM, and MuQ-Eval on CMI-RewardBench.

```bash
python -m eval.cmi_rewardbench --checkpoint checkpoints/tunejury.pt --bench-root /path/to/CMI-RewardBench --prompt-protocol with
```

--------------------------------

### Retrain single leave-out variant and batch retrain all leave-out variants

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Retrains a single leave-out variant (example: drop MusicPrefs) and then batch retrains all 6 leave-out variants. The batch script trains 6 single-leave-out (MA, MP, AIME, SE) and 2 double-leave-out (SE+MA, MP+MA) variants, each a single-seed reproducer.

```bash
# Single retrain (example: drop MusicPrefs)
python -m tunejury.train \
    --features-dir data/processed_features \
    --train-ids data/splits/train_no_MP.txt \
    --val-ids data/splits/val.txt \
    --out checkpoints/tunejury_leave_MP.pt
```

```bash
# Batch retrain all 6 leave-out variants in one go
python -m tunejury.batch_ablations \
    --features-dir data/processed_features \
    --splits-dir data/splits \
    --out-dir checkpoints/
```

--------------------------------

### Run decomposition probe with all stages

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Runs the four-stage probe (post-hoc PAM, post-hoc MusicEval, cross-distribution, stratified, scaling) via --stage all. Requires the TuneJury checkpoint and CMI-RewardBench data.

```bash
python -m eval.decomposition_probe \
    --checkpoint     checkpoints/tunejury.pt \
    --cmi-root       /path/to/CMI-RewardBench/data \
    --stage          all \
    --n-seeds-stab   20 \
    --n-seeds-scale  5 \
    --output-json    decomposition_probe.json
```

--------------------------------

### Run MuQ-Eval baseline (paper-exact and re-inference)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Two commands: the first reads precomputed summary JSON; the second re-infers from raw audio. Requires the CMI-RewardBench data path and a CUDA device.

```bash
# Paper-exact: reads applications/baselines/results/muqeval_a1/summary.json
python applications/baselines/muqeval_a1.py

# Fresh re-inference from raw audio:
python applications/baselines/muqeval_a1.py \
    --reinfer --bench_root /path/to/CMI-RewardBench/data \
    --device cuda:0
```

--------------------------------

### Reproduce probe analyses (per-system, vocal, popularity, soundfont, SingMOS, SVCC)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/applications/probes/README.md

Commands to reproduce the probe analyses described in Appendix F and Appendix I. Each probe writes a summary.json to its --out_dir and prints headline numbers to stdout. The SingMOS-Pro validation requires a 4-way shard inference step followed by an aggregation step.

```bash
python applications/probes/per_system_ranking.py \
    --features_dir data/processed_features \
    --splits_dir   data/splits \
    --meta_root    /path/to/music-ranknet/data/processed \
    --checkpoint   checkpoints/tunejury.pt \
    --out_dir      applications/probes/results/per_system
```

```bash
python applications/probes/vocal_discrimination.py \
    --features_dir data/processed_features \
    --meta_root    /path/to/music-ranknet/data/processed \
    --checkpoint   checkpoints/tunejury.pt \
    --out_dir      applications/probes/results/vocal_probe
```

```bash
python applications/probes/popularity_probe.py \
    --scores_csv  release_scores/fma_large_scores_rescored.csv \
    --listens_csv /path/to/fma_listens.csv \
    --out_dir     applications/probes/results/popularity_probe
```

```bash
python applications/probes/soundfont_sensitivity.py \
    --midi_dir   /path/to/midicaps_midi_first_300 \
    --sf_fluidr3 /usr/share/sounds/sf2/FluidR3_GM.sf2 \
    --sf_timgm6  $(python -c "import pretty_midi, os; print(os.path.join(os.path.dirname(pretty_midi.__file__), 'TimGM6mb.sf2'))") \
    --checkpoint checkpoints/tunejury.pt \
    --out_root   applications/probes/results/soundfont_probe
```

```bash
for s in 0 1 2 3; do
  python applications/probes/singmos_validation.py \
      --checkpoint   checkpoints/tunejury.pt \
      --singmos_root /path/to/SingMOS-Pro \
      --device       cuda:$s \
      --shard $s --n_shards 4 \
      --out_dir      applications/probes/results/singmos_pro
done
```

```bash
python applications/probes/singmos_validation.py \
    --aggregate --out_dir applications/probes/results/singmos_pro
```

```bash
python applications/probes/svcc_validation.py \
    --checkpoint checkpoints/tunejury.pt \
    --svcc_root  /path/to/svcc2025 \
    --out_dir    applications/probes/results/svcc2025
```

--------------------------------

### Encoder swap evaluation command (Table encoder_swap)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Runs the encoder swap evaluation using a MuQ-MuLan checkpoint. The encoder stack is detected from the checkpoint; no --feature-stack flag. Requires the muq package (pip install muq). The MuQScorer truncates audio to 120 s; shortening to 9 s drops MusicEval SRCC from 0.7848 to 0.7574.

```bash
python -m eval.cmi_rewardbench \
    --checkpoint checkpoints/tunejury_muq_leave_MA.pt \
    --bench-root /path/to/CMI-RewardBench

```

--------------------------------

### Required directory layout for CMI-RewardBench

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

The --bench-root must point to a directory containing manifest.json and an audio/ subdirectory with the four splits: pam, musiceval, cmi_pref, and music_arena. Audio paths in the manifest can be relative to bench-root or absolute.

```text
<bench-root>/
  manifest.json                # see schema below
  audio/                       # any layout; manifest paths can be relative to <bench-root> or absolute
    pam/...
    musiceval/...
    cmi_pref/...
    music_arena/...

```

--------------------------------

### Head-to-head evaluation command (Table head_to_head)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Runs the head-to-head evaluation using the T+A protocol by default. Expected results with --prompt-protocol with: PAM 0.6100, MusicEval 0.6687, CMI-Pref 0.7140, MA bench 0.7194. With --prompt-protocol empty, expected results are PAM 0.6731, MusicEval 0.6618, CMI-Pref 0.7240, MA bench 0.7007.

```bash
python -m eval.cmi_rewardbench \
    --checkpoint checkpoints/tunejury.pt \
    --bench-root /path/to/CMI-RewardBench \
    --prompt-protocol with

```

--------------------------------

### Run MuQ-Eval baseline (paper-exact or fresh re-inference)

Source: https://github.com/yonghyunk1m/tunejury/blob/main/applications/baselines/README.md

Run the MuQ-Eval baseline. The default command reads the committed summary.json and prints the four split-level numbers. Pass --reinfer with --bench_root to clone the upstream repo, download weights, and re-score all clips end-to-end.

```bash
# Paper-exact (default; reads committed summary.json):
python applications/baselines/muqeval_a1.py

# Fresh re-inference from raw audio:
python applications/baselines/muqeval_a1.py \
    --reinfer --bench_root /path/to/CMI-RewardBench/data \
    --device cuda:0
```

--------------------------------

### Recompute Delta-CLAP for SAO-small

Source: https://github.com/yonghyunk1m/tunejury/blob/main/applications/mode2_ditto/README.md

Runs compute_clap_delta.py on the SAO-small results directory. Both backbones write {pid}_baseline.wav and {pid}_ditto.wav pairs plus a summary.json directly in --out_dir.

```bash
python compute_clap_delta.py \
    --results_dir  results/ditto_sao_small \
    --json         results/ditto_sao_small/summary.json \
    --prompts_json ../../eval/prompts/sdd100.json \
    --clap_ckpt    ../../checkpoints/music_audioset_epoch_15_esc_90.14.pt \
    --limit 30
```

--------------------------------

### Download LAION-CLAP-Music checkpoint manually

Source: https://github.com/yonghyunk1m/tunejury/blob/main/checkpoints/README.md

Pre-download the LAION-CLAP-Music checkpoint (~2.2 GB) to avoid automatic download on first use. The command uses wget to fetch the file from the Hugging Face Hub.

```bash
wget https://huggingface.co/lukewys/laion_clap/resolve/main/music_audioset_epoch_15_esc_90.14.pt \
     -O music_audioset_epoch_15_esc_90.14.pt
```

--------------------------------

### Run anchor calibration experiment

Source: https://github.com/yonghyunk1m/tunejury/blob/main/README.md

Reproduce the anchor calibration results from section 6 / §A.D: ~100 pairs recover ~5 pp OOD agreement, and K=10 anchor matches K=250 retrain (~25× data-efficient on the swept K grid).

```bash
python applications/anchor_calibration/run_experiment.py
```

--------------------------------

### Generate calibration bins and render reliability diagram

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Two-step process: first emits the 10-bin reliability JSON (also re-prints accuracy/ECE/n_non_tie), then renders the calibration figure (Figure 4). Reproduces tab:calibration_bins and fig:calibration.

```bash
# Step 1: bin data (also re-prints accuracy / ECE / n_non_tie from §4.1)
python -m eval.internal \
    --checkpoint checkpoints/tunejury.pt \
    --features-dir data/processed_features \
    --test-ids data/splits/test.txt \
    --out-bins-json results/calibration_bins.json
```

```bash
# Step 2: render Figure `fig:calibration` (Figure 4 in the paper)
python figures/make_calibration_figure.py \
    --bins-json results/calibration_bins.json \
    --out       figures/fig_calibration.pdf
```

--------------------------------

### Clone repository and create conda environment

Source: https://github.com/yonghyunk1m/tunejury/blob/main/docs/reproducing.md

Commands to clone the TuneJury repository, navigate into it, create the conda environment from environment.yml, and activate it.

```bash
git clone https://github.com/yonghyunk1m/TuneJury
cd TuneJury
conda env create -f environment.yml
conda activate tunejury
```
Tokens:
Raw