MiDashengLM

https://github.com/xiaomi-research/dasheng-lm
Admin
MiDashengLM is an efficient audio language model designed for holistic audio understanding using...
Tokens:32,051
Snippets:214
Trust Score:8.4
License:Apache-2.0
Update:3 days ago
Show doc for...
Tokens:
Raw
### Install mdl-toolkit

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/README.md

Commands to set up a virtual environment and install the toolkit via pip.

```bash
# Create and activate a dedicated virtual environment with uv
uv venv path/to/mdl-toolkit-venv
source path/to/mdl-toolkit-venv/bin/activate
# Or, use venv
python -m venv path/to/mdl-toolkit-venv
source path/to/mdl-toolkit-venv/bin/activate
# Or, use conda/mamba
mamba create -n mdl-toolkit python=3.13 pip
mamba activate mdl-toolkit

# Install mdl-toolkit
pip install mdl-toolkit
# Or, if you need optional features
pip install 'mdl-toolkit[modelscope,quantization]'

# You can now use the mdl-toolkit command
mdl-toolkit --help
```

--------------------------------

### Multi-node accelerate configuration

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/distributed.md

Example YAML configuration for a 2-node setup with 8 GPUs each.

```yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
enable_cpu_affinity: false
gpu_ids: all
machine_rank: 0
main_process_ip: 10.0.0.1
main_process_port: 29500
main_training_function: main
mixed_precision: 'no'
num_machines: 2
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

--------------------------------

### Install mdl-toolkit from local source

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/installation.md

Install the package from a locally cloned repository or extracted source archive.

```bash
# Clone the repo
git clone https://github.com/xiaomi-research/dasheng-lm.git
# Or download and extract the source archive

# Create and activate a virtual environment
uv venv path/to/mdl-toolkit-venv
source path/to/mdl-toolkit-venv/bin/activate

# Install mdl-toolkit
pip install 'mdl-toolkit @ ./dasheng-lm/mdl-toolkit'
```

--------------------------------

### Create virtual environment and install mdl-toolkit

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/installation.md

Set up a virtual environment using uv, venv, or conda/mamba, then install the package with pip.

```bash
# Using uv
uv venv path/to/mdl-toolkit-venv
source path/to/mdl-toolkit-venv/bin/activate
# Or, using venv
python -m venv path/to/mdl-toolkit-venv
source path/to/mdl-toolkit-venv/bin/activate
# Or, using conda/mamba
mamba create -n mdl-toolkit python=3.13 pip
mamba activate mdl-toolkit

# Install mdl-toolkit
pip install mdl-toolkit
# Or, with optional features
pip install 'mdl-toolkit[modelscope,quantization]'

# Now you can use mdl-toolkit
mdl-toolkit --help
```

--------------------------------

### Install mdl-toolkit from Git source

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/installation.md

Install the development version directly from the GitHub repository using a VCS URL.

```bash
# Using uvx
uvx --from 'mdl-toolkit @ git+https://github.com/xiaomi-research/dasheng-lm.git#subdirectory=mdl-toolkit' mdl-toolkit --help

# Or, create and activate a virtual environment first
uv venv path/to/mdl-toolkit-venv
source path/to/mdl-toolkit-venv/bin/activate
# Then install with pip
pip install 'mdl-toolkit @ git+https://github.com/xiaomi-research/dasheng-lm.git#subdirectory=mdl-toolkit'
```

--------------------------------

### Single-node multi-GPU accelerate configuration

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/distributed.md

Example YAML configuration for a single-node setup with 8 GPUs.

```yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
enable_cpu_affinity: false
gpu_ids: all
machine_rank: 0
main_training_function: main
mixed_precision: 'no'
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

--------------------------------

### Install vLLM for Deployment

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/README.md

Commands to install vLLM either via a Python-only build or a full compilation build.

```bash
# Set up using Python-only build (without compilation)
git clone https://github.com/vllm-project/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 pip install --editable .

# Full build (with compilation)
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .
```

--------------------------------

### Start Training with mdl-toolkit

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/README.md

Basic command to initiate model training using specified dataset paths and an output directory.

```bash
mdl-toolkit train \
    --train-dataset path/to/converted/train/ \
    --eval-dataset path/to/converted/eval/ \
    --output path/to/output/
```

--------------------------------

### Training output logs

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Example output showing model loading, parameter counts, and training loss metrics.

```text
Output:
Distributed: NO
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Loading checkpoint shards: 100%|██████████████████| 4/4 [00:04<00:00,  1.01s/it]
Model loaded with torch.bfloat16
trainable params: 68,968,448 || all params: 8,350,708,352 || trainable%: 0.8259
Peak VRAM during loading: 15.684 GiB
  0%|                                                   | 0/200 [00:00<?, ?it/s]`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...
`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
{'loss': 3.8798, 'grad_norm': 3.2964117527008057, 'learning_rate': 0.0001, 'epoch': 0.01}
{'loss': 3.6867, 'grad_norm': 3.0418500900268555, 'learning_rate': 9.95e-05, 'epoch': 0.01}
{'loss': 3.1348, 'grad_norm': 2.080457925796509, 'learning_rate': 9.900000000000001e-05, 'epoch': 0.01}
{'loss': 2.8754, 'grad_norm': 2.0675864219665527, 'learning_rate': 9.850000000000001e-05, 'epoch': 0.02}
{'loss': 2.5002, 'grad_norm': 2.108095169067383, 'learning_rate': 9.8e-05, 'epoch': 0.03}
{'loss': 1.9693, 'grad_norm': 2.2269351482391357, 'learning_rate': 9.75e-05, 'epoch': 0.03}
{'loss': 1.6182, 'grad_norm': 2.1482107639312744, 'learning_rate': 9.7e-05, 'epoch': 0.04}
{'loss': 1.4915, 'grad_norm': 1.8752493858337402, 'learning_rate': 9.65e-05, 'epoch': 0.04}
{'loss': 1.3654, 'grad_norm': 1.7094244956970215, 'learning_rate': 9.6e-05, 'epoch': 0.04}
{'loss': 1.0916, 'grad_norm': 1.8022851943969727, 'learning_rate': 9.55e-05, 'epoch': 0.05}
{'loss': 0.9587, 'grad_norm': 2.0512678623199463, 'learning_rate': 9.5e-05, 'epoch': 0.06}
{'loss': 0.6848, 'grad_norm': 2.383748769760132, 'learning_rate': 9.449999999999999e-05, 'epoch': 0.06}
{'loss': 0.6857, 'grad_norm': 1.3146827220916748, 'learning_rate': 9.4e-05, 'epoch': 0.07}
{'loss': 0.4695, 'grad_norm': 1.1016651391983032, 'learning_rate': 9.350000000000001e-05, 'epoch': 0.07}
{'loss': 0.4135, 'grad_norm': 1.3678333759307861, 'learning_rate': 9.300000000000001e-05, 'epoch': 0.07}
{'loss': 0.4895, 'grad_norm': 1.2987140417099, 'learning_rate': 9.250000000000001e-05, 'epoch': 0.08}
{'loss': 0.5315, 'grad_norm': 1.2354402542114258, 'learning_rate': 9.200000000000001e-05, 'epoch': 0.09}
{'loss': 0.4365, 'grad_norm': 1.5113264322280884, 'learning_rate': 9.15e-05, 'epoch': 0.09}
{'loss': 0.3974, 'grad_norm': 1.2639994621276855, 'learning_rate': 9.1e-05, 'epoch': 0.1}
{'loss': 0.3503, 'grad_norm': 0.8667172789573669, 'learning_rate': 9.05e-05, 'epoch': 0.1}
{'loss': 0.4957, 'grad_norm': 1.751583456993103, 'learning_rate': 9e-05, 'epoch': 0.1}
{'loss': 0.3468, 'grad_norm': 1.2601685523986816, 'learning_rate': 8.950000000000001e-05, 'epoch': 0.11}
{'loss': 0.4162, 'grad_norm': 1.393071174621582, 'learning_rate': 8.900000000000001e-05, 'epoch': 0.12}
{'loss': 0.3701, 'grad_norm': 0.8844708204269409, 'learning_rate': 8.850000000000001e-05, 'epoch': 0.12}
{'loss': 0.3662, 'grad_norm': 1.160038948059082, 'learning_rate': 8.800000000000001e-05, 'epoch': 0.12}
```

--------------------------------

### Verify MDL-Toolkit installation

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Check if MDL-Toolkit is installed by displaying the help message.

```python
# Install MDL-Toolkit, for example:
# !pip install mdl-toolkit
!mdl-toolkit --help
```

--------------------------------

### Run vLLM Inference and Serving

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/README.md

Execute offline inference or start an OpenAI-compatible API server using vLLM.

```bash
# Offline inference
python3 examples/offline_inference/audio_language.py -m midashenglm

# Online serving using OpenAI-compatible server
python3 -m vllm.entrypoints.openai.api_server --model mispeech/midashenglm-7b --tensor-parallel-size 1 --served-model-name default --port 8000 --dtype float16 --max_model_len 4096 --trust_remote_code
```

--------------------------------

### Install Evaluation Dependencies

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/README.md

Install the required Python packages for evaluation. This step is not required for model inference.

```bash
pip install -r requirements.txt
```

--------------------------------

### ESC-50 Dataset Format Example

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/README.md

Example of a basic CSV dataset format for audio samples.

```csv
audio,prediction
audio/1-100032-A-0.wav,"target: 0, category: dog"
audio/1-100038-A-14.wav,"target: 14, category: chirping_birds"
audio/1-100210-A-36.wav,"target: 36, category: vacuum_cleaner"
```

--------------------------------

### ESC-50 Dataset with Prompts

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/README.md

Example of a CSV dataset including optional system and user prompts.

```csv
audio,system_prompt,user_prompt,prediction
audio/1-100032-A-0.wav,null,What is the sound in the audio?,It sounds like a dog barking.
audio/1-100038-A-14.wav,Classify the audio according to the ESC-50 categories.,null,chirping_birds
audio/1-100210-A-36.wav,Answer user's question about the audio.,Is that a vacuum cleaner?,Yes.
```

--------------------------------

### Inference Output Logs

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Example output logs generated during the model loading and inference process.

```text
Output:
/home/h30/zhoujiahao5/notebook/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:15<00:00,  3.78s/it]
```

```text
Output:
['category: pig, target: 2']
```

--------------------------------

### Convert datasets for training

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Converts training and testing CSV files into a format optimized for training, applying a system prompt to guide model behavior.

```python
!mdl-toolkit convert-dataset \
    train.csv \
    --output train-converted/ \
    --system-prompt "Output the predicted category in the format of category: <category>, category_id: <category_id>."
!mdl-toolkit convert-dataset \
    test.csv \
    --output test-converted/ \
    --system-prompt "Output the predicted category in the format of category: <category>, category_id: <category_id>."
```

--------------------------------

### Distributed Training with accelerate

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/README.md

Commands to configure and launch distributed training using the accelerate library.

```bash
accelerate config # Follow the interactive prompts
accelerate launch \
    mdl-toolkit train \
    --train-dataset path/to/converted/train/ \
    --eval-dataset path/to/converted/eval/ \
    --output path/to/output/
```

--------------------------------

### Distributed Training with torchrun (Single Node)

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/README.md

Command to launch distributed training on a single node using torchrun.

```bash
torchrun --standalone --nproc-per-node gpu --no-python \
    mdl-toolkit train \
    --train-dataset path/to/converted/train/ \
    --eval-dataset path/to/converted/eval/ \
    --output path/to/output/
```

--------------------------------

### Configure accelerate environment

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/distributed.md

Interactively configure the distributed training environment for accelerate.

```bash
accelerate config
```

--------------------------------

### Tune Hyperparameters via CLI

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/README.md

Example of overriding default training hyperparameters such as learning rate and LoRA rank.

```bash
mdl-toolkit train \
    --lr 1e-4 \
    --lora-rank 32 \
    ...
```

--------------------------------

### Distributed Training with torchrun (Multi-Node)

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/README.md

Command to launch distributed training across multiple nodes using torchrun with rendezvous configuration.

```bash
torchrun --nnodes $NUM_NODES --nproc-per-node gpu \
    --rdzv-id $JOB_ID \
    --rdzv-backend c10d \
    --rdzv-endpoint $HOST_NODE_ADDR \
    --no-python \
    mdl-toolkit train \
    --train-dataset path/to/converted/train/ \
    --eval-dataset path/to/converted/eval/ \
    --output path/to/output/
```

--------------------------------

### Single-node training with torchrun

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/distributed.md

Use this command to utilize all available GPUs on a single node in standalone mode.

```bash
torchrun --standalone --nproc-per-node gpu --no-python \
    mdl-toolkit train \
    --lora-rank 16 \
    --eval-steps 50 \
    --train-dataset train-converted/ \
    --eval-dataset test-converted/ \
    --output output/
```

--------------------------------

### Run mdl-toolkit with uvx

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/installation.md

Execute mdl-toolkit in an isolated environment using uvx, optionally including the modelscope and quantization features.

```bash
uvx mdl-toolkit --help
# Or, with optional features
uvx --from 'mdl-toolkit[modelscope,quantization]' mdl-toolkit --help
```

--------------------------------

### Verify training and test splits

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Prints the first 11 lines of the generated training and test CSV files to verify the split and formatting.

```python
!echo '==== Train split ===='
!head -n 11 train.csv
!echo '==== Test split  ===='
!head -n 11 test.csv
```

--------------------------------

### Run inference with mdl-toolkit

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Executes inference on test.csv using the specified model and system prompt, then displays the first 11 lines of the output.

```python
!mdl-toolkit inference \
    test.csv \
    --system-prompt "Output the predicted category in the format of category: <category>, category_id: <category_id>." \
    --output orig-output.csv \
    --model-name mispeech/midashenglm-7b-bf16
! head -n 11 orig-output.csv
```

--------------------------------

### Compare Original and Fine-tuned Outputs

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Displays the first 11 lines of the original and fine-tuned CSV output files to compare predictions.

```bash
! head -n 11 orig-output.csv
! head -n 11 finetuned-output.csv
```

--------------------------------

### MDL-Toolkit help output

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

The expected output when running the help command for mdl-toolkit.

```text
Output:
usage: mdl-toolkit [-h] {train,convert-dataset,inference} ...

options:
  -h, --help            show this help message and exit

subcommands:
  {train,convert-dataset,inference}
    train
    convert-dataset
    inference
```

--------------------------------

### Load MiDashengLM Model

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/README.md

Initialize the model, tokenizer, and processor using the Hugging Face Auto classes. Requires setting trust_remote_code to True.

```python
from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer

# For 7B:
model_id = "mispeech/midashenglm-7b-1021-bf16"
# For 0.6B:
# model_id = "mispeech/midashenglm-0.6b-fp32"

model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
```

--------------------------------

### Train the model with mdl-toolkit

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Execute the training process using the specified LoRA rank, dataset paths, and model name.

```python
!mdl-toolkit train \
     --lora-rank 32 \
     --eval-steps 100 \
     --train-dataset train-converted/ \
     --eval-dataset test-converted/ \
     --output output/ \
     --model-name mispeech/midashenglm-7b-bf16
```

--------------------------------

### Perform Inference with Python

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/esc-50.ipynb

Loads a fine-tuned model and processor to generate predictions for an audio file. Set TOKENIZERS_PARALLELISM to 0 to avoid warnings in Notebook environments.

```python
# Set TOKENIZERS_PARALLELISM=0 in the Notebook to prevent warnings from
# messing up the output. Do not set this outside the Notebook, as it may
# cause reduced performance.
import os

os.environ["TOKENIZERS_PARALLELISM"] = "0"

import torch
from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer

model_id = "./output/final/"

model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

model.eval()

messages = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "Output the predicted category in the format of category: <category>, category_id: <category_id>.",
            },
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "audio", "path": "ESC-50-master/audio/5-103415-A-2.wav"},
        ],
    },
]

with torch.no_grad():
    model_inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        add_special_tokens=True,
        return_dict=True,
    ).to(device=model.device, dtype=model.dtype)
    generation = model.generate(**model_inputs)
    output = tokenizer.batch_decode(generation, skip_special_tokens=True)

print(output)
```

--------------------------------

### Specify accelerate config file

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/mdl-toolkit/docs_en/distributed.md

Use a specific configuration file path for accelerate.

```bash
accelerate config --config_file /path/to/config/file
```

--------------------------------

### Execute Evaluation Scripts

Source: https://github.com/xiaomi-research/dasheng-lm/blob/main/README.md

Run the provided Python scripts to evaluate model outputs against specific metrics like WER, ACC, mAP, and FENSE for various audio tasks.

```bash
# Automatic Speech Recognition (WER)
# Uses: lang, text, model_output
python evaluate/wer/compute_wer.py -i evaluate/jsonl/MiDashengLM_LibriSpeech_test-clean.jsonl

# Single-target Audio Tagging (ACC)
# Uses: label, model_output
python evaluate/compute_at_acc.py -i evaluate/jsonl/MiDashengLM_NSynth.jsonl

# Gender Recognition (ACC)
# Uses: label, model_output
python evaluate/compute_gender_acc.py -i evaluate/jsonl/MiDashengLM_VoxCeleb-Gender.jsonl

# Multi-target Audio Tagging (mAP)
# Uses: dataset_name, label, model_output, model_name
python evaluate/compute_map.py -i evaluate/jsonl/MiDashengLM_FSD50K.jsonl

# Audio Captioning (FENSE)
# Uses: audio, text, model_output
python evaluate/compute_fense.py -i evaluate/jsonl/MiDashengLM_AutoACD.jsonl

# Open Audio QA (FENSE)
# Uses: audio, answer, model_output
python evaluate/compute_fense.py -i evaluate/jsonl/MiDashengLM_MusicQA.jsonl

# Audio QA with Options (ACC)
# Uses: answer, model_output
python evaluate/compute_qa_acc.py -i evaluate/jsonl/MiDashengLM_MuChoMusic.jsonl
```
Tokens:
Raw