SLAM-LLM

https://github.com/x-lance/slam-llm
Admin
SLAM-LLM is a deep learning toolkit for training custom multimodal large language models focused on...
Tokens:34,210
Snippets:272
Trust Score:7.3
License:MIT
Update:10 months ago
Show doc for...
Tokens:
Raw
### Environment Setup for SELD_SpatialSoundQA

Source: https://github.com/x-lance/slam-llm/blob/main/examples/seld_spatialsoundqa/README.md

Installs necessary dependencies for the SELD_SpatialSoundQA example. It involves changing directories, installing from a requirements file, and then installing the main project package.

```bash
cd SLAM-LLM/examples/seld_spatialsoundqa/
pip install -r requirements.txt
cd SLAM-LLM/
pip install -e .
```

--------------------------------

### Install Fairseq and SLAM-LLM

Source: https://github.com/x-lance/slam-llm/blob/main/README.md

Instructions to clone the fairseq repository, navigate into the directory, and install it as an editable package. This is a prerequisite for using SLAM-LLM.

```shell
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./
```

--------------------------------

### Install Matcha TTS from Source

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/README.md

Installs Matcha TTS directly from its GitHub repository. This method is useful for developers who want to contribute to the project or use the latest unreleased features. It requires cloning the repository and installing it in editable mode.

```bash
pip install git+https://github.com/shivammehta25/Matcha-TTS.git
cd Matcha-TTS
pip install -e .
```

--------------------------------

### Install ONNX Runtime for Inference

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/README.md

Installs the ONNX Runtime library, necessary for running inference on exported ONNX models. Includes options for CPU and GPU inference.

```bash
pip install onnxruntime
```

```bash
pip install onnxruntime-gpu  # for GPU inference
```

--------------------------------

### Install Dependencies with Pip

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/README.md

Installs the necessary Python packages for the project using a requirements file. Assumes the SLAM-LLM environment is already prepared.

```bash
pip install -r ./examples/s2s/requirements.txt
```

--------------------------------

### Install ONNX for Export

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/README.md

Installs the ONNX package, a prerequisite for exporting Matcha checkpoints to the ONNX format. This enables model deployment on various platforms.

```bash
pip install onnx
```

--------------------------------

### Install Matcha TTS via Pip

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/README.md

Installs the Matcha TTS library using pip. This is the recommended method for users who want to quickly integrate the TTS system. Dependencies like Python 3.10, PyTorch 2.0+, and Lightning 2.0+ are typically required.

```bash
pip install matcha-tts
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/x-lance/slam-llm/blob/main/examples/st_covost2/README.md

Installs the necessary Python packages and system dependencies for the SLAM-LLM project, including PyTorch, Transformers, and Whisper. Ensures the environment is set up for speech translation tasks.

```bash
conda create -n cotst python=3.10
conda activate cotst

git clone https://github.com/ddlBoJack/SLAM-LLM.git
cd SLAM-LLM

pip install -e .
sudo apt install ffmpeg
pip install -U openai-whisper
pip install wandb
pip install soundfile
pip install evaluate
pip install transformers
pip install datasets
pip install sacrebleu
pip install jiwer
pip install librosa
pip install torch==2.4.0
pip install torchaudio==2.4.0
pip install torchvision==0.19.0
```

--------------------------------

### Example JSONL Data Format

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/README.md

Illustrates the structure of a single data entry in JSONL format, used for dialogue modeling. Includes source audio path, text, and target audio tokens/text.

```json
{"key": "1", "source_wav": "/xxx/1.wav", "source_text": "Can you recommend some Chinese food for me?", "target_token": [742, 383, 455, 619, 180], "target_text": "Sure! I recommend trying dumplings, Peking duck, and mapo tofu for a mix of flavors and textures in Chinese cuisine. These dishes offer a good balance of savory, spicy, and crispy elements."}
```

--------------------------------

### Start Matcha TTS Gradio App

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/README.md

Launches the Gradio web interface for Matcha TTS, providing an interactive way to test the speech synthesis functionality. This command also handles the automatic download of required pre-trained models.

```bash
matcha-tts-app
```

--------------------------------

### Prepare Data in JSONL Format

Source: https://github.com/x-lance/slam-llm/blob/main/examples/st_covost2/README.md

Provides an example of the required JSONL data format for ST_covost2. Each entry includes an audio file path, a prompt for translation, the ground truth text, and the source of the data. This format is crucial for training and evaluation.

```json
{"audio": "/userhome/speech/data/common/4/en/clips/common_voice_en_699711.mp3", "prompt": "<|en|>", "gt": "\"She'll be all right.\"", "source": "covost_en"}
{"audio": "/userhome/speech/data/common/4/en/clips/common_voice_en_699711.mp3", "prompt": "<|de|>", "gt": "\"She'll be all right.\"<|de|>Sie wird schon in Ordnung sein.", "source": "covost_ende"}
{"audio": "/userhome/speech/data/common/4/en/clips/common_voice_en_699711.mp3", "prompt": "<|ja|>", "gt": "\"She'll be all right.\"<|ja|>彼女は大丈夫だろう。", "source": "covost_enja"}
{"audio": "/userhome/speech/data/common/4/en/clips/common_voice_en_699711.mp3", "prompt": "<|zh|>", "gt": "\"She'll be all right.\"<|zh|>她会没事的。", "source": "covost_enzh"}
{"audio": "/userhome/speech/data/common/4/en/clips/common_voice_en_699711.mp3", "prompt": "\"She'll be all right.\"<|de|>". "gt": "\"She'll be all right.\"<|de|>Sie wird schon in Ordnung sein.", "source": "covost_enende"}
{"audio": "/userhome/speech/data/common/4/en/clips/common_voice_en_699711.mp3", "prompt": "\"She'll be all right.\"<|ja|>". "gt": "\"She'll be all right.\"<|ja|>彼女は大丈夫だろう。", "source": "covost_enenja"}
{"audio": "/userhome/speech/data/common/4/en/clips/common_voice_en_699711.mp3", "prompt": "\"She'll be all right.\"<|zh|>". "gt": "\"She'll be all right.\"<|zh|>她会没事的。", "source": "covost_enenzh"}
```

--------------------------------

### DeepSpeed Configuration for Training

Source: https://github.com/x-lance/slam-llm/blob/main/examples/asr_librispeech/README.md

Example DeepSpeed configuration file (conf/ds_config.json) for training, enabling FP16 mixed precision and ZeRO stage 2 optimization with optimizer offloading to CPU. This configuration can save significant GPU memory during training.

```json
{
    "train_micro_batch_size_per_gpu": 4,
    "gradient_accumulation_steps": 1,
    "optimizer": {
        "type": "Adam",
        "params": {
            "lr": 1e-4
        }
    },
    "fp16": {
        "enabled": true
    },
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "cpu"
        }
    }
}
```

--------------------------------

### Pre-train TTS Model using Bash Script

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/scripts/pretrain/README.md

This script initiates the pre-training process for a Text-to-Speech (TTS) model. It assumes data is prepared in the required format (Parquet or JSONL). The task involves learning to generate target speech from given text.

```bash
bash ./examples/s2s/scripts/pretrain/pretrain_tts.sh
```

--------------------------------

### Multiprompt Data Format Example (JSON)

Source: https://github.com/x-lance/slam-llm/blob/main/examples/aispeech_asr/README.md

Illustrates the format for multiprompt.jsonl, which defines various prompts for different tasks like ASR, translation, and hotword detection. The prompt string can dynamically include information using placeholders like '{}'.

```json
{"task": "ASR", "prompt": "Transcribe speech to text."}
{"task": "ASR", "prompt": "请识别语音."}
{"task": "ZH2EN", "prompt": "请识别语音并翻译为英文:"}
{"task": "EN2ZH", "prompt": "请识别语音并翻译为中文:"}
{"task": "prevtext", "prompt": "Transcribe speech to text, below are the previous historical transcription texts:{}."}
{"task": "hotword", "prompt": "Transcribe speech to text, follow words may occur:{}."}
```

--------------------------------

### Pre-train ASR Model using Bash Script

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/scripts/pretrain/README.md

This script is used to pre-train an Automatic Speech Recognition (ASR) model. It requires data to be in the specified format and focuses on the task of generating text transcripts from given speech.

```bash
bash ./examples/s2s/scripts/pretrain/pretrain_asr.sh
```

--------------------------------

### Online TTS Inference using Bash Script

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/scripts/pretrain/README.md

This script enables online inference for testing Text-to-Speech (TTS) models. It allows you to generate speech in real-time using the pre-trained TTS models.

```bash
bash ./examples/s2s/scripts/inference/inference_tts_online.sh
```

--------------------------------

### Online ASR Inference using Bash Script

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/scripts/pretrain/README.md

This script facilitates online inference for testing Automatic Speech Recognition (ASR) models. It is used to generate text transcripts from speech in real-time with pre-trained ASR models.

```bash
bash ./examples/s2s/scripts/inference/inference_asr_online.sh
```

--------------------------------

### Import Necessary Libraries for Matcha-TTS

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/synthesis.ipynb

Imports essential libraries for the Matcha-TTS project, including PyTorch for deep learning, NumPy for numerical operations, soundfile for audio handling, and custom modules from the Matcha-TTS library for model definition, text processing, and utilities. This setup is crucial for loading models and performing synthesis.

```python
import datetime as dt
from pathlib import Path

import IPython.display as ipd
import numpy as np
import soundfile as sf
import torch
from tqdm.auto import tqdm

# Hifigan imports
from matcha.hifigan.config import v1
from matcha.hifigan.denoiser import Denoiser
from matcha.hifigan.env import AttrDict
from matcha.hifigan.models import Generator as HiFiGAN
# Matcha imports
from matcha.models.matcha_tts import MatchaTTS
from matcha.text import sequence_to_text, text_to_sequence
from matcha.utils.model import denormalize
from matcha.utils.utils import get_user_data_dir, intersperse
```

--------------------------------

### Multitask Data Format Example (JSON)

Source: https://github.com/x-lance/slam-llm/blob/main/examples/aispeech_asr/README.md

Defines the structure for multitask.jsonl, used for preparing ASR and other related tasks. It specifies key, task, target transcription, and the path to audio data, supporting both ark format and direct wav files.

```json
{"key": "BAC009S0002W0122", "task": "ASR", "target": "而对楼市成交抑制作用最大的限购", "path": "/aistor/aispeech/hpc_stor01/group/asr/mandarin/aishell-1/asr/train/data/data_wav.1.ark:17"}
{"key": "BAC009S0002W0123", "task": "ASR", "target": "也成为地方政府的眼中钉", "path": "/aistor/aispeech/hpc_stor01/group/asr/mandarin/aishell-1/asr/train/data/data_wav.1.ark:191758"}
{"key": "BAC009S0002W0124", "task": "ASR", "target": "自六月底呼和浩特市率先宣布取消限购后", "path": "/aistor/aispeech/hpc_stor01/group/asr/mandarin/aishell-1/asr/train/data/data_wav.1.ark:315339"}
{"key": "BAC009S0764W0238", "task": "hotword", "path": "/aistor/aispeech/hpc_stor01/group/asr/mandarin/aishell-1/asr/test/data/data_wav.1.ark:17343733", "target": "形成一批具有国际竞争力的中国企业", "hotword": "中国"}
```

--------------------------------

### Distributed Training with FSDP and Torchrun (Bash)

Source: https://context7.com/x-lance/slam-llm/llms.txt

This bash script demonstrates how to launch a distributed fine-tuning job across multiple GPUs using `torchrun` and FSDP. It sets environment variables for CUDA and OpenMP, defines training parameters, and uses `torchrun` to distribute the `finetune_asr.py` script across specified nodes and processes per node. Configuration overrides are passed as command-line arguments.

```bash
#!/bin/bash

# Set environment variables
export CUDA_VISIBLE_DEVICES=0,1,2,3
export OMP_NUM_THREADS=1

# Training configuration
model_name="vicuna-7b-v1.5"
speech_encoder_path="large-v2"
llm_path="/path/to/vicuna-7b-v1.5"
train_data_path="/data/train.jsonl"
val_data_path="/data/val.jsonl"
output_dir="./exp/whisper_vicuna_7b_linear"

# Run training with torchrun
torchrun \
    --nnodes=1 \
    --nproc_per_node=4 \
    --master_port=29503 \
    finetune_asr.py \
    --config-name "prompt" \
    ++model_config.llm_name="${model_name}" \
    ++model_config.llm_path="${llm_path}" \
    ++model_config.encoder_name=whisper \
    ++model_config.encoder_path="${speech_encoder_path}" \
    ++model_config.encoder_projector=linear \
    ++model_config.encoder_dim=1280 \
    ++model_config.llm_dim=4096 \
    ++dataset_config.dataset_name=speech_dataset \
    ++dataset_config.train_data_path="${train_data_path}" \
    ++dataset_config.val_data_path="${val_data_path}" \
    ++dataset_config.input_type=mel \
    ++dataset_config.mel_size=80 \
    ++train_config.model_name="${model_name}" \
    ++train_config.enable_fsdp=true \
    ++train_config.enable_ddp=false \
    ++train_config.num_epochs=3 \
    ++train_config.batch_size=2 \
    ++train_config.gradient_accumulation_steps=8 \
    ++train_config.lr=1e-4 \
    ++train_config.warmup_steps=1000 \
    ++train_config.total_steps=50000 \
    ++train_config.freeze_encoder=true \
    ++train_config.freeze_llm=false \
    ++train_config.use_peft=true \
    ++train_config.output_dir="${output_dir}" \
    ++log_config.use_wandb=true \
    ++log_config.wandb_project_name="slam-llm-asr"

# Expected output:
# [2024-04-01 10:00:00][slam_llm.pipeline.finetune][INFO] - Training started
# [2024-04-01 10:00:30][slam_llm.pipeline.finetune][INFO] - Epoch 1/3, Step 100/50000, Loss: 2.345, Acc: 0.456
# [2024-04-01 10:15:00][slam_llm.pipeline.finetune][INFO] - Validation Loss: 2.123, WER: 15.6%
```

--------------------------------

### Initialize SLAM-LLM Model and Components

Source: https://github.com/x-lance/slam-llm/blob/main/examples/seld_spatialsoundqa/inference.ipynb

Initializes the SLAM-LLM model, including training and model configurations, and loads pre-trained weights. It sets up the model and tokenizer, moves the model to the appropriate device (GPU or CPU), and sets it to evaluation mode.

```python
from omegaconf import OmegaConf
from seld_config import TrainConfig, ModelConfig
from model.slam_model_seld import model_factory

train_config = TrainConfig(
    model_name="BAT",
    batching_strategy="custom",
    num_epochs=1,
    num_workers_dataloader=2,
    use_peft=True,
    freeze_encoder=True,
    freeze_llm=True
)
train_config = OmegaConf.merge(train_config)

model_config = ModelConfig(
    llm_name="llama-2-7b",
    llm_path="https://huggingface.co/meta-llama/Llama-2-7b-hf", # 
    encoder_name="SpatialAST",
    encoder_ckpt="https://huggingface.co/datasets/zhisheng01/SpatialAudio/blob/main/SpatialAST/finetuned.pth", # 
)

kwargs = {
    "decode_log": None,
    "ckpt_path": "https://huggingface.co/datasets/zhisheng01/SpatialAudio/blob/main/BAT/model.pt", # Download it from huggingface
}
model, tokenizer = model_factory(train_config, model_config, **kwargs)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # FIX(MZY): put the whole model to device.
model.to(device)
model.eval()
```

--------------------------------

### Inference on AudioCaps using CLAP-Refine

Source: https://github.com/x-lance/slam-llm/blob/main/examples/slam_aac/README.md

This command executes the CLAP-Refine strategy for inference on the AudioCaps dataset. This method aims to generate improved audio captions by refining existing candidates, utilizing a pre-trained CLAP model.

```bash
bash scripts/inference_audiocaps_CLAP_Refine.sh
```

--------------------------------

### Define Text for Synthesis

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/synthesis.ipynb

Defines a list of text strings that will be synthesized into speech. This serves as the input data for the speech synthesis pipeline. The example provides a single sentence to be converted into audio.

```python
texts = [
    "The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent."
]
```

--------------------------------

### Load and Inspect Parquet Files with Datasets Library (Python)

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/demo/demo_data/demo.ipynb

This Python code illustrates how to load Parquet files using the `datasets` library, specifying multiple files. It then prints the resulting `DatasetDict` object, showing the structure, features, and number of rows in the dataset. This is useful for a quick look at the data format.

```python
from datasets import load_dataset

parquet_files = [
    "pq_demo-zh.parquet",
    "pq_demo-en.parquet",
]
d = load_dataset('parquet', data_files=parquet_files)
d
```

--------------------------------

### Conv1D Projection Layer with Downsampling

Source: https://context7.com/x-lance/slam-llm/llms.txt

Utilizes `EncoderProjectorCov1d` to apply a Conv1D projection layer that also downsamples the input sequence. This is useful for reducing sequence length while projecting features. The example configures downsampling rate and demonstrates its application.

```python
from slam_llm.models.projector import EncoderProjectorCov1d

class ProjectorConfig:
    encoder_dim = 1280
    llm_dim = 4096
    encoder_projector = "cov1d-linear"
    encoder_projector_ds_rate = 5  # Downsample rate

config = ProjectorConfig()
projector = EncoderProjectorCov1d(config)

# Downsample and project
encoder_output = torch.randn(4, 500, 1280)  # [batch, 500, encoder_dim]
projected = projector(encoder_output)  # [batch, 100, llm_dim]
print(f"Downsampled 5x: {projected.shape}")  # [4, 100, 4096]

```

--------------------------------

### Build and Run SLAM-LLM Docker Image

Source: https://github.com/x-lance/slam-llm/blob/main/README.md

Commands to build a Docker image for SLAM-LLM and run it with GPU access. The Docker image provides a convenient environment for using the project's functionalities.

```shell
# build docker image
docker build -t slam-llm:latest .

# run docker image with gpu
docker run -it --gpus all --name slam --shm-size=256g slam-llm:latest /bin/bash
```

--------------------------------

### Fine-tuning with WavLM Large Linear Vicuna 7B

Source: https://github.com/x-lance/slam-llm/blob/main/examples/asr_librispeech/README.md

Bash script for fine-tuning the SLAM-ASR model using a self-supervised model like WavLM as the encoder and Vicuna 7B as the LLM. WavLM models accept raw waveforms. Pay attention to `dataset_config.normalize` and `model_config.normalize` which may vary for different SSL models.

```bash
finetune_wavlm_large_linear_vicuna_7b.sh
```

--------------------------------

### Run Matcha TTS CLI for Synthesis

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/README.md

Executes the Matcha TTS command-line interface to synthesize speech from input text. This command will automatically download necessary pre-trained models. Options are available for specifying text, file input, batch synthesis, speaking rate, sampling temperature, and ODE solver steps.

```bash
# Synthesize from text
matcha-tts --text "<INPUT TEXT>"

# Synthesize from a file
matcha-tts --file <PATH TO FILE>

# Batch synthesize from a file
matcha-tts --file <PATH TO FILE> --batched

# With additional arguments
matcha-tts --text "<INPUT TEXT>" --speaking_rate 1.0 --temperature 0.667 --steps 10
```

--------------------------------

### Configure Autoreload and Matplotlib

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/synthesis.ipynb

Enables autoreload for Python modules, allowing changes to be reflected immediately without restarting the kernel. Matplotlib is configured for inline plotting, which is standard for Jupyter notebooks. This setup enhances the development workflow by providing real-time updates and integrated visualizations.

```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
```

--------------------------------

### Inference on Clotho using CLAP-Refine

Source: https://github.com/x-lance/slam-llm/blob/main/examples/slam_aac/README.md

This script applies the CLAP-Refine strategy for inference on the Clotho dataset. It refines audio caption candidates using a pre-trained CLAP model to enhance caption quality.

```bash
bash scripts/inference_clotho_CLAP_Refine.sh
```

--------------------------------

### Download and Save Parquet Dataset with Datasets Library (Python)

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/demo/demo_data/demo.ipynb

This code snippet demonstrates how to download a dataset from the Hugging Face Hub in Parquet format using the `datasets` library. It also shows how to save the downloaded dataset to disk for local access and subsequently load it back from the saved path.

```python
from datasets import load_dataset, load_from_disk

# Download the dataset
ds = load_dataset("gpt-omni/VoiceAssistant-400K")

# Save the dataset to disk if needed
save_path = "/path/to/save/directory"
ds.save_to_disk(save_path)

# Load the dataset from disk if needed
ds = load_from_disk(save_path)
```

--------------------------------

### Q-Former Projection Layer for Fixed-Length Features

Source: https://context7.com/x-lance/slam-llm/llms.txt

Employs `EncoderProjectorQFormer` to extract a fixed number of query tokens from variable-length encoder outputs. This Q-Former approach is beneficial for advanced feature extraction and alignment with LLMs. The example shows configuration and usage with an attention mask.

```python
from slam_llm.models.projector import EncoderProjectorQFormer

class ProjectorConfig:
    encoder_dim = 1280
    llm_dim = 4096
    encoder_projector = "q-former"
    num_query_tokens = 32  # Fixed number of queries

config = ProjectorConfig()
projector = EncoderProjectorQFormer(config)

# Extract fixed-length features
encoder_output = torch.randn(4, 500, 1280)  # Variable length
attention_mask = torch.ones(4, 500)
projected = projector(encoder_output, attention_mask)  # [batch, 32, llm_dim]
print(f"Fixed query output: {projected.shape}")  # [4, 32, 4096]

```

--------------------------------

### Data Preparation Format for SpatialSoundQA

Source: https://github.com/x-lance/slam-llm/blob/main/examples/seld_spatialsoundqa/README.md

Illustrates the required JSONL format for preparing data for the SpatialSoundQA dataset. Each entry contains audio identifiers, question details, and the corresponding answer.

```json
{
  "audio_id": "eval/audio/YI-HlrcP6Qg4",
  "reverb_id": "q9vSo1VnCiC/0.npy", 
  "audio_id2": null,
  "reverb_id2": null,
  "question_id": 0,
  "question_type": "CLASSIFICATION",
  "question": "Enumerate the sound occurrences in the audio clip.",
  "answer": "accelerating, revving, vroom; car; vehicle"
}

...

{
  "audio_id": "eval/audio/YZX2fVPmUidA", 
  "reverb_id": "q9vSo1VnCiC/32.npy", 
  "audio_id2": "eval/audio/YjNjUU01quLs", 
  "reverb_id2": "q9vSo1VnCiC/31.npy", 
  "question_id": 58,
  "question_type": "MIXUP_NONBINARY_DISTANCE", 
  "question": "How far away is the sound of the banjo from the sound of the whack, thwack?", 
  "answer": "2m"
}
```

--------------------------------

### Utilize Pre-trained Audio Encoders (Whisper, WavLM, BEATs, CLAP)

Source: https://context7.com/x-lance/slam-llm/llms.txt

Demonstrates how to load and use pre-trained audio encoders provided by the Slam-LLM library. It covers Whisper for mel spectrograms, WavLM and BEATs for raw waveforms, and CLAP for audio-text alignment. Input and output dimensions for each are specified.

```python
from slam_llm.models.encoder import (
    WhisperWrappedEncoder,
    WavLMEncoder,
    BEATsEncoder,
    CLAPEncoder
)

# Whisper encoder for mel spectrogram input
whisper_encoder = WhisperWrappedEncoder.load(model_config)
# Input: [batch, n_mels=80, time]
# Output: [batch, time//2, 1280]

# WavLM encoder for raw waveform
wavlm_encoder = WavLMEncoder.load(model_config)
# Input: [batch, samples]
# Output: [batch, frames, 1024]

# BEATs encoder for audio understanding
beats_encoder = BEATsEncoder.load(model_config)
# Input: [batch, samples]
# Output: [batch, frames, 768]

# CLAP encoder for audio-text alignment
clap_encoder = CLAPEncoder.load(model_config)
# Can encode both audio and text to same embedding space
# audio_embeds = clap_encoder.encode_audio(audio)
# text_embeds = clap_encoder.encode_text(text)

```

--------------------------------

### Linear Projection Layer for Encoder Output

Source: https://context7.com/x-lance/slam-llm/llms.txt

Implements a linear projection layer using `EncoderProjectorConcat` to map the output dimensions of an encoder (e.g., Whisper's 1280) to the input dimensions of an LLM (e.g., Vicuna-7B's 4096). The example shows creating the projector and applying it to sample encoder output.

```python
from slam_llm.models.projector import EncoderProjectorConcat

class ProjectorConfig:
    encoder_dim = 1280  # Whisper large output dim
    llm_dim = 4096  # Vicuna-7B input dim
    encoder_projector = "linear"

config = ProjectorConfig()
projector = EncoderProjectorConcat(config)

# Project encoder output to LLM input space
encoder_output = torch.randn(4, 100, 1280)  # [batch, seq, encoder_dim]
projected = projector(encoder_output)  # [batch, seq, llm_dim]
print(f"Projected shape: {projected.shape}")  # [4, 100, 4096]

```

--------------------------------

### Run DRCap Model Training Script

Source: https://github.com/x-lance/slam-llm/blob/main/examples/drcap_zeroshot_aac/README.md

Initiates the training process for the DRCap model. Optional parameters allow for training only the linear layer, freezing the LLM, or disabling Retrieval-Augmented Generation (RAG).

```shell
bash scripts/finetune_drcap.sh
```

--------------------------------

### Speech Dataset Management in JSONL Format (Python)

Source: https://context7.com/x-lance/slam-llm/llms.txt

This Python code demonstrates how to prepare and load speech data using the SpeechDatasetJsonl class. It shows how to create a JSONL file containing audio file paths, transcriptions, and their lengths. It also illustrates initializing the dataset with a configuration and tokenizer, and then creating a DataLoader for training.

```python
from slam_llm.datasets.speech_dataset import SpeechDatasetJsonl
import json

# Prepare training data in JSONL format
train_data = [
    {
        "key": "example_001",
        "source": "/data/audio/example_001.wav",
        "target": "This is the transcription of the audio file.",
        "source_len": 480000,  # Audio length in samples
        "target_len": 42  # Token length
    },
    {
        "key": "example_002",
        "source": "/data/audio/example_002.wav",
        "target": "Another example transcription.",
        "source_len": 320000,
        "target_len": 28
    }
]

# Write to JSONL
with open("/data/train.jsonl", "w") as f:
    for item in train_data:
        f.write(json.dumps(item) + "\n")

# Load dataset
from dataclasses import dataclass

@dataclass
class DatasetConfig:
    train_data_path: str = "/data/train.jsonl"
    val_data_path: str = "/data/val.jsonl"
    input_type: str = "mel"
    mel_size: int = 80
    normalize: bool = False
    prompt: str = "Transcribe speech to text."
    fix_length_audio: int = -1
    inference_mode: bool = False

dataset_config = DatasetConfig()

# Create dataset instance
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("/path/to/vicuna-7b-v1.5")
tokenizer.pad_token_id = tokenizer.eos_token_id

dataset = SpeechDatasetJsonl(
    dataset_config=dataset_config,
    tokenizer=tokenizer,
    split="train"
)

# Get a sample
sample = dataset[0]
print(f"Input shape: {sample['input_ids'].shape}")
print(f"Labels shape: {sample['labels'].shape}")
print(f"Audio mel shape: {sample['audio_mel'].shape}")

# Create DataLoader with custom collator
from torch.utils.data import DataLoader

dataloader = DataLoader(
    dataset,
    batch_size=4,
    shuffle=True,
    num_workers=4,
    collate_fn=dataset.collator
)

# Iterate through batches
for batch in dataloader:
    input_ids = batch["input_ids"]  # [batch_size, seq_len]
    labels = batch["labels"]  # [batch_size, seq_len]
    audio_mel = batch["audio_mel"]  # [batch_size, mel_len, mel_dim]
    modality_mask = batch["modality_mask"]  # [batch_size, seq_len]
    print(f"Batch input_ids: {input_ids.shape}")
    break
```

--------------------------------

### Add Custom Audio Encoder with PyTorch

Source: https://context7.com/x-lance/slam-llm/llms.txt

Defines a `CustomAudioEncoder` class using PyTorch's `nn.Module` for audio feature extraction. It includes convolutional layers and a Transformer encoder. The `load` class method handles loading from a checkpoint, and the `forward` method processes audio input. Registration and usage examples are provided.

```python
import torch
import torch.nn as nn

class CustomAudioEncoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        # Define your encoder architecture
        self.conv_layers = nn.Sequential(
            nn.Conv1d(80, 256, kernel_size=3, stride=2),
            nn.ReLU(),
            nn.Conv1d(256, 512, kernel_size=3, stride=2),
            nn.ReLU()
        )
        self.transformer = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=512, nhead=8),
            num_layers=6
        )

    @classmethod
    def load(cls, model_config):
        """Factory method for loading encoder"""
        encoder = cls(model_config)
        if hasattr(model_config, 'encoder_path'):
            checkpoint = torch.load(model_config.encoder_path)
            encoder.load_state_dict(checkpoint)
        return encoder

    def forward(self, audio_input, padding_mask=None):
        """
        Args:
            audio_input: [batch_size, time, features]
            padding_mask: [batch_size, time]
        Returns:
            encoder_output: [batch_size, seq_len, hidden_dim]
        """
        x = self.conv_layers(audio_input.transpose(1, 2))
        x = x.transpose(1, 2)
        x = self.transformer(x)
        return x

# Register encoder in slam_llm/models/encoder.py
# Add to setup_encoder function:
# if encoder_name == "custom_encoder":
#     from slam_llm.models.encoder import CustomAudioEncoder
#     encoder = CustomAudioEncoder.load(model_config)

# Use in training
model_config.encoder_name = "custom_encoder"
model_config.encoder_path = "/path/to/custom_encoder.pt"
model_config.encoder_dim = 512

```

--------------------------------

### Model Training Script for AAC_Audiocaps

Source: https://github.com/x-lance/slam-llm/blob/main/examples/aac_audiocaps/README.md

This bash script initiates the fine-tuning process for the AAC_Audiocaps model. Users can customize paths for the audio encoder, LLM, output directory, and data files. Specific parameters like 'use_peft' and 'freeze_llm' control whether PEFT methods are employed and if the LLM is frozen during training.

```bash
bash scripts/finetune_eat_audiocaps.sh
```

--------------------------------

### Prepare Audio Data for SLAM-LLM in Python

Source: https://github.com/x-lance/slam-llm/blob/main/examples/seld_spatialsoundqa/inference.ipynb

This snippet demonstrates how to prepare audio data for the SLAM-LLM model. It initializes audio representation tokens, converts a NumPy waveform to a PyTorch tensor, and applies padding to a fixed length. Dependencies include PyTorch and NumPy.

```python
audio_length = 64 # We use 64 learnable tokens as audio representation
audio_pseudo = torch.full((audio_length,), -1)
waveform = torch.from_numpy(waveform).float()
waveform = SpatialAudioDatasetJsonl.padding(waveform, padding_length=10*32000-waveform.shape[1])
```

--------------------------------

### Format Prompts for LLM Querying

Source: https://github.com/x-lance/slam-llm/blob/main/examples/seld_spatialsoundqa/inference.ipynb

Formats a list of natural language questions about the audio into a specific prompt structure expected by the LLM. It uses a helper function `format_prompt` and prints the formatted prompts.

```python
prompts = [
    "What is the distance between the sound of the drum and the sound of the siren?",
    "What is the sound on the right side of the sound of the drum?",
    "Are you able to detect the percussion's sound coming from the left and the emergency vehicle's sounds from the right?",
]

gt_answers = [
    "1.5m",
    "emergency vehicle; fire engine, fire truck (siren); siren",
    "Yes"
]

prompts = [format_prompt(prompt) for prompt in prompts]
print(prompts)
```

--------------------------------

### Directly Refine Audio Caption Candidates with CLAP-Refine

Source: https://github.com/x-lance/slam-llm/blob/main/examples/slam_aac/README.md

This command allows for direct refinement of already generated audio caption candidates using the CLAP-Refine strategy. It takes existing candidates as input and applies the refinement process without re-running the full inference.

```bash
bash scripts/clap_refine.sh
```

--------------------------------

### Train Model using MusicFM Encoder

Source: https://github.com/x-lance/slam-llm/blob/main/examples/mc_musiccaps/README.md

Shell script to initiate the training process for a new model, utilizing MusicFM as the encoder and Vicuna 7B as the LLM. This script is intended for fine-tuning or training from scratch.

```bash
#!/bin/bash

# Example usage:
# bash finetune_musicfm_linear_vicuna_7b_10s.sh

# Modify these paths and parameters according to your setup:
train_data_path="path/to/your/train_data.jsonl"
val_data_path="path/to/your/val_data.jsonl"
output_dir="path/to/save/trained/model"
music_encoder_path="path/to/musicfm/encoder.pt"
music_encoder_stat_path="path/to/musicfm/encoder.stat"
ckpt_path="path/to/vicuna-7b-v1.5"

# The actual training command would go here, using the variables above.
# For example:
# python train.py \
#   --train_data_path $train_data_path \
#   --val_data_path $val_data_path \
#   --output_dir $output_dir \
#   --music_encoder_path $music_encoder_path \
#   --music_encoder_stat_path $music_encoder_stat_path \
#   --ckpt_path $ckpt_path \
#   --learning_rate 1e-5 \
#   --num_train_epochs 3
```

--------------------------------

### Inference on Clotho with Beam Search

Source: https://github.com/x-lance/slam-llm/blob/main/examples/slam_aac/README.md

This script performs inference on the Clotho dataset using beam search decoding for generating audio captions. It leverages a pre-trained CLAP model to achieve higher quality results, though it may take longer to execute.

```bash
bash scripts/inference_clotho_bs.sh
```

--------------------------------

### Train Matcha-TTS Model

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/README.md

Provides commands to initiate the training process for the Matcha-TTS model on the LJSpeech dataset. Supports standard training, minimum memory training, and multi-GPU training.

```bash
make train-ljspeech
```

```bash
python matcha/train.py experiment=ljspeech
```

```bash
python matcha/train.py experiment=ljspeech_min_memory
```

```bash
python matcha/train.py experiment=ljspeech trainer.devices=[0,1]
```

--------------------------------

### Pull and Run Docker Image

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/README.md

Pulls a pre-built Docker image for SLAM-Omni and runs it with GPU access. This provides a self-contained environment for the project.

```bash
docker pull worstchan/slam-omni:v0
docker run -it --gpus all --name slam-omni worstchan/slam-omni:v0 /bin/bash
```

--------------------------------

### Load and Play Anechoic Audio Files

Source: https://github.com/x-lance/slam-llm/blob/main/examples/seld_spatialsoundqa/inference.ipynb

Loads two `.wav` audio files and plays them using IPython's `Audio` display. This snippet demonstrates basic audio file loading and playback, providing context for the audio content through print statements.

```python
audio_path1 = "./assets/YCqvbWnTBfTk.wav"
audio_path2 = "./assets/Yq4Z8j3IalYs.wav"

print("Let's load and listen to anechoic audio...")
print("Audio 1: Drum; Percussion")
display(Audio(audio_path1))

print("Audio 2: Emergency vehicle; Fire engine, fire truck (siren); Siren")
display(Audio(audio_path2))
```

--------------------------------

### Execute Training Stages for Speech Translation

Source: https://github.com/x-lance/slam-llm/blob/main/examples/st_covost2/README.md

Initiates the training process for ST_covost2. The first script performs Automatic Speech Recognition (ASR) pretraining. The second script, 'all.sh', handles monolingual Multimodal Machine Translation (MMT), Speech Recognition (SRT) training, and multitask training, allowing task type modification via the 'source' parameter.

```bash
#In this step, we perform ASR pretraining to acquire speech recognition capabilities.
bash examples/st_covost2/scripts/asr_pretrain.sh


#monolingual MMT,SRT training and multitask training. 
#You can change the task type by modifying the value of **source** in the script.
bash examples/st_covost2/scripts/all.sh
```

--------------------------------

### Data Preparation JSONL Format

Source: https://github.com/x-lance/slam-llm/blob/main/examples/mc_musiccaps/README.md

Specifies the required JSONL format for data preparation in the MC_MusicCaps project. Each entry should include a unique key, the source path to the audio file, the target caption, and optionally, the duration and sample rate for efficiency.

```json
{
"key": "[-0Gj8-vB1q4]-[30-40]", 
"source": "path/to/MusicCaps/wav/[-0Gj8-vB1q4]-[30-40].wav", 
"target": "The low quality recording features a ballad song that contains sustained strings, mellow piano melody and soft female vocal singing over it. It sounds sad and soulful, like something you would hear at Sunday services.", 
"duration": 10.0, 
"sample_rate": 48000
}
...
{
"key": "[-0vPFx-wRRI]-[30-40]", 
"source": "path/to/MusicCaps/wav/[-0vPFx-wRRI]-[30-40].wav", 
"target": "a male voice is singing a melody with changing tempos while snipping his fingers rhythmically. The recording sounds like it has been recorded in an empty room. This song may be playing, practicing snipping and singing along.", 
"duration": 10.0, 
"sample_rate": 48000
}
```

--------------------------------

### Train HiFi-GAN Model

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/matcha/hifigan/README.md

This command initiates the training process for the HiFi-GAN model. It requires a configuration file specifying training parameters. Different configuration files (e.g., config_v1.json, config_v2.json) can be used to train different versions of the generator.

```python
python train.py --config config_v1.json
```

--------------------------------

### Download Pre-trained Models for Speech Translation

Source: https://github.com/x-lance/slam-llm/blob/main/examples/st_covost2/README.md

Clones the repositories for the pre-trained Whisper-large-v3 encoder, a q-former projector, and the Qwen2-7B LLM. These models are essential components for the ST_covost2 speech translation system.

```bash
git lfs clone https://huggingface.co/openai/whisper-large-v3
git lfs clone https://huggingface.co/yxdu/cotst
git lfs clone https://huggingface.co/Qwen/Qwen2-7B
```

--------------------------------

### Run DRCap Data Preprocessing Script

Source: https://github.com/x-lance/slam-llm/blob/main/examples/drcap_zeroshot_aac/README.md

This script performs retrieval-augmentation and creates text embeddings necessary for evaluating the DRCap model. It processes the prepared data to support evaluation metrics.

```shell
bash scripts/data_preprocess.sh
```

--------------------------------

### Train New Model using finetune_spatial-ast_qformer_llama_2_7b.sh

Source: https://github.com/x-lance/slam-llm/blob/main/examples/seld_spatialsoundqa/README.md

Executes a bash script to finetune the Spatial-AST model with Q-Former and Llama-2-7b. This is part of the training process for the BAT model.

```bash
cd examples/seld_spatialsoundqa/
bash scripts/finetune_spatial-ast_qformer_llama_2_7b.sh
```

--------------------------------

### Run Inference Demo for English to Chinese Translation

Source: https://github.com/x-lance/slam-llm/blob/main/examples/st_covost2/README.md

Executes the inference demo script for English to Chinese speech translation. This command downloads models and datasets automatically and requires significant resources (100GB storage, 128GB RAM, 24GB GPU memory per card). Supports Chinese (zh), German (de), and Japanese (ja) translation.

```bash
CUDA_VISIBLE_DEVICES=0 bash examples/st_covost2/scripts/infer_enzh.sh zh
```

--------------------------------

### Online Inference for Multi-Round Dialogue

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/README.md

Initiates the online inference process for multi-round dialogue scenarios. This script is recommended for real-time interaction.

```bash
bash ./examples/s2s/scripts/inference/inference_s2s_online_multi-round.sh
```

--------------------------------

### Data Preparation Format for AAC_Audiocaps

Source: https://github.com/x-lance/slam-llm/blob/main/examples/aac_audiocaps/README.md

This snippet shows the required JSONL format for preparing data for the AAC_Audiocaps model. Each entry must include a 'key', 'source' path to the audio file, and a 'target' description. Ensure your data adheres to this structure for successful model training.

```json
{"key": "Y7fmOlUlwoNg_1", "source": "/root/data/AudioCaps/waveforms/test/Y7fmOlUlwoNg.wav", "target": "Constant rattling noise and sharp vibrations"}
{"key": "Y6BJ455B1aAs_1", "source": "/root/data/AudioCaps/waveforms/test/Y6BJ455B1aAs.wav", "target": "A rocket flies by followed by a loud explosion and fire crackling as a truck engine runs idle"}
```

--------------------------------

### Synthesize Audio Waveforms with SLAM-LLM (Python)

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/synthesis.ipynb

This Python code snippet iterates through a list of texts, synthesizes audio waveforms using a specified model, and calculates the Real-Time Factor (RTF) for both the synthesis and the full waveform generation process. It also includes pretty printing of input text, phonetized text, RTF values, and displays/saves the generated waveform. Dependencies include `tqdm`, `torch`, `datetime`, `IPython.display`, and custom functions like `synthesise`, `to_waveform`, and `save_to_folder`.

```python
temperature = 0.667

outputs, rtfs = [], []
rtfs_w = []
for i, text in enumerate(tqdm(texts)):
    output = synthesise(text) #, torch.tensor([15], device=device, dtype=torch.long).unsqueeze(0))
    output['waveform'] = to_waveform(output['mel'], vocoder)

    # Compute Real Time Factor (RTF) with HiFi-GAN
    t = (dt.datetime.now() - output['start_t']).total_seconds()
    rtf_w = t * 22050 / (output['waveform'].shape[-1])

    ## Pretty print
    print(f"{'*' * 53}")
    print(f"Input text - {i}")
    print(f"{'-' * 53}")
    print(output['x_orig'])
    print(f"{'*' * 53}")
    print(f"Phonetised text - {i}")
    print(f"{'-' * 53}")
    print(output['x_phones'])
    print(f"{'*' * 53}")
    print(f"RTF:\t\t{output['rtf']:.6f}")
    print(f"RTF Waveform:\t{rtf_w:.6f}")
    rtfs.append(output['rtf'])
    rtfs_w.append(rtf_w)

    ## Display the synthesised waveform
    ipd.display(ipd.Audio(output['waveform'], rate=22050))

    ## Save the generated waveform
    save_to_folder(i, output, OUTPUT_FOLDER)

print(f"Number of ODE steps: {n_timesteps}")
print(f"Mean RTF:\t\t\t\t{np.mean(rtfs):.6f} ± {np.std(rtfs):.6f}")
print(f"Mean RTF Waveform (incl. vocoder):\t{np.mean(rtfs_w):.6f} ± {np.std(rtfs_w):.6f}")
```

--------------------------------

### Decoding with WavLM Large Linear Vicuna 7B

Source: https://github.com/x-lance/slam-llm/blob/main/examples/asr_librispeech/README.md

Shell script command to initiate decoding using a pre-trained WavLM Large encoder, a linear projector, and the Vicuna 7B LLM. Users need to modify paths for speech encoder, LLM, output directory, checkpoint, validation data, and log files.

```bash
decode_wavlm_large_linear_vicuna_7b.sh
```

--------------------------------

### Synthesize Speech from Custom Model

Source: https://github.com/x-lance/slam-llm/blob/main/examples/s2s/utils/third_party/Matcha-TTS/README.md

Command-line interface to synthesize speech using a custom-trained Matcha-TTS model. Requires the input text and the path to the trained checkpoint.

```bash
matcha-tts --text "<INPUT TEXT>" --checkpoint_path <PATH TO CHECKPOINT>
```
Tokens:
Raw