MuseTok

https://github.com/yuer867/musetok
Admin
MuseTok is a symbolic music tokenization framework designed for music generation and semantic...
Tokens:7,120
Snippets:62
Trust Score:6.3
Update:4 weeks ago
Show doc for...
Tokens:
Raw
### Install dependencies

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-tokenization.ipynb

Installs required packages from requirements.txt.

```python
!pip install -r requirements.txt
```

--------------------------------

### Install dependencies with pip

Source: https://github.com/yuer867/musetok/blob/main/README.md

Install the required Python packages listed in requirements.txt. The project was tested with Python 3.10 and torch==2.5.1.

```bash
pip install -r requirements.txt
```

--------------------------------

### Load and encode MIDI file into MuseTok codes

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-generation.ipynb

Converts a MIDI file into music events and encodes them into MuseTok codes and embeddings. The example uses 'test/Classic_358.mid' and sets a primer of 4 bars. Output shows the number of bars and events, and the shapes of the codes and embeddings.

```python
from data_processing.midi2events import midi_analyzer, midi2corpus_strict, corpus2events
from encoding import MuseTokEncoder
from test_generation import generate_tokens, decode_tokens, word2event, TempoEvent
from remi2midi import remi2midi
from utils import numpy_to_tensor

file = 'test/Classic_358.mid'
primer_n_bar = 4

# represent MIDI file into music events
midi_obj, bar_resol = midi_analyzer(file)
full_data = midi2corpus_strict(midi_obj, bar_resol, remove_overlap=True)
pos, events = corpus2events(full_data, bar_resol, time_first=False, has_velocity=False, repeat_beat=True, remove_short=False)
print('[info] got {} bars, {} events'.format(len(pos), len(events)))

# encode music events
encoder = MuseTokEncoder(tokenizer, device=device)
music_data = encoder.get_segments(events, pos)
indices, latents = encoder.encoding(music_data, return_latents=True)
print('[info] Shape of MuseTok codes: {}'.format(indices.shape))
print('[info] Shape of corresponding embeddings: {}'.format(latents.shape))
```

```text
Output:
[info] Represent MIDI file test/Classic_358.mid ...
[info] time signature = 4/4
[info] got 16 bars, 999 events
[info] Shape of MuseTok codes: (16, 16)
[info] Shape of corresponding embeddings: (16, 128)
```

--------------------------------

### Decode generated codes to MIDI file

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-generation.ipynb

Decodes the generated MuseTok embeddings back into music events and writes them to a MIDI file. The example saves to 'test/continuation.mid' with a tempo of 110 BPM and 4/4 time signature. Output shows the number of events per bar and total generation time.

```python
# decode codes back to music events
song, t_sec = decode_tokens(
    tokenizer, gen_latents,
    encoder.event2idx, encoder.idx2event, device,
    max_events=12800
)
song = word2event(song, encoder.idx2event)

out_file = 'test/continuation'
midi_obj = remi2midi(
    song, 
    out_file + '.mid', 
    enforce_tempo=True, 
    enforce_tempo_val=[TempoEvent(110, 0, 0, 4, 4)]
)
```

```text
Output:
[info] generated 1 bars, #events = 50
[info] generated 2 bars, #events = 112
[info] generated 3 bars, #events = 168
[info] generated 4 bars, #events = 218
[info] generated 5 bars, #events = 274
[info] generated 6 bars, #events = 336
[info] generated 7 bars, #events = 404
[info] generated 8 bars, #events = 463
[info] generated 9 bars, #events = 546
[info] generated 10 bars, #events = 608
[info] generated 11 bars, #events = 688
[info] generated 12 bars, #events = 771
[info] generated 13 bars, #events = 836
[info] generated 14 bars, #events = 901
[info] generated 15 bars, #events = 972
[info] generated 16 bars, #events = 1025
-- generated events: 1026
-- time elapsed: 34.78 secs
```

--------------------------------

### Generate music by continuing prompts

Source: https://github.com/yuer867/musetok/blob/main/README.md

Generate music pieces by continuing prompts from test sets using the two-stage generation framework. Requires downloaded best weights and configuration file config/generation.yaml.

```bash
python test_generation.py \
        --configuration=config/generation.yaml \
        --model=ckpt/best_generator/model.pt \
        --use_prompt \
        --primer_n_bar=4 \
        --n_pieces=20 \
        --output_dir=samples/generation
```

--------------------------------

### Train music tokenization model

Source: https://github.com/yuer867/musetok/blob/main/README.md

Train a music tokenization model from scratch using the configuration file config/tokenization.yaml.

```bash
python train_tokenizer.py config/tokenization.yaml
```

--------------------------------

### Output of tokenizer loading

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-tokenization.ipynb

Confirms successful loading of the tokenizer.

```text
Output:
[info] successfully load tokenizer
```

--------------------------------

### Generate music from scratch

Source: https://github.com/yuer867/musetok/blob/main/README.md

Generate music pieces from scratch without prompts. Uses the same model and configuration as the prompt-based generation, but without the --use_prompt flag.

```bash
python test_generation.py \
        --configuration=config/generation.yaml \
        --model=ckpt/best_generator/model.pt \
        --n_pieces=20 \
        --output_dir=samples/generation
```

--------------------------------

### Build Vocabulary

Source: https://github.com/yuer867/musetok/blob/main/data_processing/README.md

Run this command to build the vocabulary from the event sequences. The script is located at data_processing/events2words.py.

```bash
python data_processing/events2words.py
```

--------------------------------

### Load pre-trained tokenizer and generator

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-generation.ipynb

Loads the pre-trained TransformerResidualVQ tokenizer and GPT2TokenGenerator from the checkpoint files. The tokenizer uses 16 quantizers and a codebook size of 2048, while the generator uses a vocabulary of 32771 tokens and 16 bars. Both models are set to evaluation mode and loaded onto the CUDA device.

```python
# load pre-trained tokenizer and generator
import torch
import numpy as np
from model.musetok import TransformerResidualVQ, GPT2TokenGenerator

import warnings
warnings.simplefilter(action='ignore')

device = 'cuda'
num_quantizers = 16
codebook_size = 2048

tokenizer_path = 'ckpt/best_tokenizer/model.pt'
tokenizer = TransformerResidualVQ(
    enc_n_layer=12, enc_n_head=8, enc_d_model=512, enc_d_ff=2048,
    dec_n_layer=12, dec_n_head=8, dec_d_model=512, dec_d_ff=2048,
    d_vae_latent=128, d_embed=512, n_token=168,
    num_quantizers=16, codebook_size=2048,
    rotation_trick=True, rvq_type='SimVQ'
).to(device)
tokenizer.eval()
tokenizer.load_state_dict(torch.load(tokenizer_path, map_location='cpu'))
print('[info] successfully load tokenizer')

generator_path = 'ckpt/best_generator/model.pt'
generator = GPT2TokenGenerator(
    dec_n_layer=12, dec_n_head=16, dec_d_model=1024, dec_d_ff=2048,
    d_embed=512, n_token=32771, n_bar=16
).to(device)
generator.eval()
generator.load_state_dict(torch.load(generator_path, map_location='cpu'))
print('[info] successfully load generator')
```

--------------------------------

### Load pre-trained tokenizer

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-tokenization.ipynb

Loads the pre-trained MuseTok tokenizer model from checkpoint.

```python
# load pre-trained tokenizer
import torch
from model.musetok import TransformerResidualVQ

import warnings
warnings.simplefilter(action='ignore')

device = 'cuda'
ckpt_path = 'ckpt/best_tokenizer/model.pt'
model = TransformerResidualVQ(
    enc_n_layer=12, enc_n_head=8, enc_d_model=512, enc_d_ff=2048,
    dec_n_layer=12, dec_n_head=8, dec_d_model=512, dec_d_ff=2048,
    d_vae_latent=128, d_embed=512, n_token=168,
    num_quantizers=16, codebook_size=2048,
    rotation_trick=True, rvq_type='SimVQ'
).to(device)
model.eval()
model.load_state_dict(torch.load(ckpt_path, map_location='cpu'))
print('[info] successfully load tokenizer')
```

--------------------------------

### Generate MuseTok codes from scratch

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-generation.ipynb

Generates MuseTok codes from scratch using a generator model with a primer token, then converts the codes to embeddings via the tokenizer's residual_sim_vq. The output shows progress messages for each generated bar and the total elapsed time.

```python
# generate MuseTok codes from scratch
gen_tokens, t_sec = generate_tokens(
    generator, primer=[bos_token], primer_n_bar=primer_n_bar,
    max_bars=16, num_tokens=num_quantizers, codebook_size=codebook_size,
    temp=1.1, top_p=0.9, top_k=30, eos=bos_token + 1
)

# transfer codes to corresponding embeddings
num_bars = len(gen_tokens) // num_quantizers
gen_tokens = np.array(gen_tokens) - np.tile(np.arange(num_quantizers), num_bars) * codebook_size
gen_indices = numpy_to_tensor(gen_tokens, device=device).view(-1, num_quantizers).long()
gen_latents = tokenizer.residual_sim_vq.get_output_from_indices(gen_indices)
```

```text
Output:
[info] generated 1 bars
[info] generated 2 bars
[info] generated 3 bars
[info] generated 4 bars
[info] generated 5 bars
[info] generated 6 bars
[info] generated 7 bars
[info] generated 8 bars
[info] generated 9 bars
[info] generated 10 bars
[info] generated 11 bars
[info] generated 12 bars
-- time elapsed: 3.15 secs
```

--------------------------------

### Train music generation model

Source: https://github.com/yuer867/musetok/blob/main/README.md

Train a music generation model using the pre-encoded tokens. Requires the configuration file config/generation.yaml.

```bash
python train_generator.py config/generation.yaml
```

--------------------------------

### Generate music with new checkpoints

Source: https://github.com/yuer867/musetok/blob/main/README.md

Generate music pieces using a new checkpoint. The --model flag specifies the checkpoint to use; change it to your own. The command uses prompts and generates 20 pieces of 4 bars each.

```bash
python test_generation.py \
        --configuration=config/generation.yaml \
        --model=ckpt/best_generator/model.pt \  # change the checkpoints here
        --use_prompt \
        --primer_n_bar=4 \
        --n_pieces=20 \
        --output_dir=samples/generation
```

--------------------------------

### Test reconstruction quality

Source: https://github.com/yuer867/musetok/blob/main/README.md

Test the reconstruction quality of a trained tokenizer using music pieces from the test sets. The command takes the configuration file, model checkpoint, output directory, and number of pieces as arguments.

```bash
python test_reconstruction.py config/tokenization.yaml ckpt/best_tokenizer/model.pt samples/reconstruction 20
```

--------------------------------

### Unzip checkpoints

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-tokenization.ipynb

Unzips the downloaded checkpoint file.

```python
# 2. unzip the file
!unzip ckpt.zip
```

--------------------------------

### Data Splits

Source: https://github.com/yuer867/musetok/blob/main/data_processing/README.md

Run this command to split the dataset into training, validation, and test sets. The script is located at data_processing/data_split.py.

```bash
python data_processing/data_split.py
```

--------------------------------

### Convert MIDI to events

Source: https://github.com/yuer867/musetok/blob/main/data_processing/README.md

Run this command to convert MIDI files into REMI event sequences. The script is located at data_processing/midi2events.py.

```bash
python data_processing/midi2events.py
```

--------------------------------

### Generate continuation MuseTok codes

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-generation.ipynb

Generates MuseTok codes for continuation using a transformer. The prompt is built from the first 4 bars of the encoded indices. The generation uses temperature 1.1, top-p 0.9, and top-k 30, and outputs 16 bars. The generated codes are then converted to embeddings.

```python
# generate MuseTok codes by continuation
indices = (indices + np.arange(num_quantizers) * codebook_size).reshape(-1).tolist()
bos_token = len(range(num_quantizers * codebook_size))
prompt = [bos_token] + indices[:primer_n_bar * num_quantizers]

gen_tokens, t_sec = generate_tokens(
    generator, primer=prompt, primer_n_bar=primer_n_bar,
    max_bars=16, num_tokens=num_quantizers, codebook_size=codebook_size,
    temp=1.1, top_p=0.9, top_k=30, eos=bos_token + 1
)

# transfer codes to corresponding embeddings
num_bars = len(gen_tokens) // num_quantizers
gen_tokens = np.array(gen_tokens) - np.tile(np.arange(num_quantizers), num_bars) * codebook_size
gen_indices = numpy_to_tensor(gen_tokens, device=device).view(-1, num_quantizers).long()
gen_latents = tokenizer.residual_sim_vq.get_output_from_indices(gen_indices)
```

```text
Output:
[info] generated 5 bars
[info] generated 6 bars
[info] generated 7 bars
[info] generated 8 bars
[info] generated 9 bars
[info] generated 10 bars
[info] generated 11 bars
[info] generated 12 bars
[info] generated 13 bars
[info] generated 14 bars
[info] generated 15 bars
[info] generated 16 bars
-- time elapsed: 3.18 secs
```

--------------------------------

### Output: Tqdm warning

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-generation.ipynb

Output from the previous code block showing a TqdmWarning about IProgress not found, suggesting to update jupyter and ipywidgets.

```text
Output:
/home/jingyue/miniconda3/envs/mmp/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
```

--------------------------------

### Represent MIDI file with REMI+ events

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-tokenization.ipynb

Converts a MIDI file to REMI+ events. Supports piano music with constant time signatures 2/4, 3/4, 4/4, 2/2, 3/8, 6/8, defaulting to 4/4 if not provided.

```python
from data_processing.midi2events import midi_analyzer, midi2corpus_strict, corpus2events

file = 'test/Classic_358.mid'
midi_obj, bar_resol = midi_analyzer(file)
full_data = midi2corpus_strict(midi_obj, bar_resol, remove_overlap=True)
pos, events = corpus2events(full_data, bar_resol, time_first=False, has_velocity=False, repeat_beat=True, remove_short=False)
print('[info] got {} bars, {} events'.format(len(pos), len(events)))
```

--------------------------------

### Output: Success messages

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-generation.ipynb

Output from the previous code block showing successful loading of the tokenizer and generator.

```text
Output:
[info] successfully load tokenizer
[info] successfully load generator
```

--------------------------------

### Output of encoding

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-tokenization.ipynb

Shows the shapes of the resulting MuseTok codes and embeddings.

```text
Output:
[info] Shape of MuseTok codes: (73, 16)
[info] Shape of corresponding embeddings: (73, 128)
```

--------------------------------

### Output of MIDI event extraction

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-tokenization.ipynb

Shows the output of the MIDI event extraction, including time signature and event count.

```text
Output:
[info] Represent MIDI file test/Classic_358.mid ...
[info] time signature = 4/4
[info] got 16 bars, 999 events
```

--------------------------------

### Decode codes to MIDI with remi2midi

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-generation.ipynb

Decodes the generated latent codes back into music events using the tokenizer and encoder, then converts the events to a MIDI file with a specified tempo. The output shows the number of events generated per bar and the total elapsed time.

```python
# decode codes back to music events
song, t_sec = decode_tokens(
    tokenizer, gen_latents,
    encoder.event2idx, encoder.idx2event, device,
    max_events=12800
)
song = word2event(song, encoder.idx2event)

out_file = 'test/generation'
midi_obj = remi2midi(
    song, 
    out_file + '.mid', 
    enforce_tempo=True, 
    enforce_tempo_val=[TempoEvent(110, 0, 0, 4, 4)]
)
```

```text
Output:
[info] generated 1 bars, #events = 8
[info] generated 2 bars, #events = 25
[info] generated 3 bars, #events = 42
[info] generated 4 bars, #events = 56
[info] generated 5 bars, #events = 70
[info] generated 6 bars, #events = 87
[info] generated 7 bars, #events = 107
[info] generated 8 bars, #events = 127
[info] generated 9 bars, #events = 135
[info] generated 10 bars, #events = 152
[info] generated 11 bars, #events = 169
[info] generated 12 bars, #events = 186
-- generated events: 187
-- time elapsed: 2.12 secs
```

--------------------------------

### Encode music events

Source: https://github.com/yuer867/musetok/blob/main/MuseTok-tokenization.ipynb

Encodes music events into MuseTok codes and embeddings using the pre-trained encoder.

```python
# encode music events
from encoding import MuseTokEncoder
encoder = MuseTokEncoder(model, device=device)

music_data = encoder.get_segments(events, pos)
indices, latents = encoder.encoding(music_data, return_latents=True)
print('[info] Shape of MuseTok codes: {}'.format(indices.shape))
print('[info] Shape of corresponding embeddings: {}'.format(latents.shape))
```

--------------------------------

### Encode REMI to RVQ tokens

Source: https://github.com/yuer867/musetok/blob/main/README.md

Encode REMI sequences to RVQ tokens offline with data augmentation. This step is optional if you use tokens encoded with provided tokenizer weights and have downloaded the datasets.

```bash
python remi2tokens.py config/remi2tokens.yaml ckpt/best_tokenizer/model.pt
```
Tokens:
Raw