### Install SGLang Backend

Source: https://context7.com/openmoss/moss-tts/llms.txt

Setup commands for the SGLang accelerated inference backend.

```bash
# Clone and install SGLang
git clone https://github.com/OpenMOSS/sglang.git
pip install -e ./sglang/python[all]
```

--------------------------------

### Install training dependencies

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_local/finetuning/README.md

Standard installation for the MossTTSLocal training environment.

```bash
git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,finetune]"
```

--------------------------------

### MOSS-TTS Inference Setup and Generation

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

This code demonstrates how to set up the MOSS-TTS model, processor, and generate audio outputs. It includes examples for direct TTS, pinyin/IPA input, voice cloning with reference audio, and duration control using token counts. Ensure audio files are accessible locally to avoid cloud downloads.

```python
ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"
ref_audio_2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a"

conversations = [
    # Direct TTS (no reference)
    [processor.build_user_message(text=text_1)],
    [processor.build_user_message(text=text_2)],
    # Pinyin or IPA input
    [processor.build_user_message(text=text_3)],
    [processor.build_user_message(text=text_4)],
    [processor.build_user_message(text=text_5)],
    [processor.build_user_message(text=text_6)],
    # Voice cloning (with reference)
    [processor.build_user_message(text=text_1, reference=[ref_audio_1])],
    [processor.build_user_message(text=text_2, reference=[ref_audio_2])],
    # Duration control
    [processor.build_user_message(text=text_2, tokens=325)],
    [processor.build_user_message(text=text_2, tokens=600)],
]

model = AutoModel.from_pretrained(
    pretrained_model_name_or_path,
    trust_remote_code=True,
    attn_implementation=attn_implementation,
    torch_dtype=dtype,
).to(device)
model.eval()

batch_size = 1

save_dir = Path("inference_root")
save_dir.mkdir(exist_ok=True, parents=True)
sample_idx = 0
with torch.no_grad():
    for start in range(0, len(conversations), batch_size):
        batch_conversations = conversations[start : start + batch_size]
        batch = processor(batch_conversations, mode="generation")
        input_ids = batch["input_ids"].to(device)
        attention_mask = batch["attention_mask"].to(device)

        outputs = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            max_new_tokens=4096,
        )

        for message in processor.decode(outputs):
            audio = message.audio_codes_list[0]
            out_path = save_dir / f"sample{sample_idx}.wav"
            sample_idx += 1
            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)
```

--------------------------------

### Install Training Dependencies

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/finetuning/README.md

Standard installation for training dependencies. Use the extra-index-url for CUDA 12.8 support.

```bash
git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,finetune]"
```

```bash
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,finetune-deepspeed]"
```

--------------------------------

### Install MOSS-TTS-Delay dependencies

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/llama_cpp/README_zh.md

Choose the installation command based on your hardware acceleration requirements.

```bash
pip install -e ".[llama-cpp-onnx]"
```

```bash
pip install -e ".[llama-cpp-trt]"
```

```bash
pip install -e ".[llama-cpp-onnx,llama-cpp-torch]"
```

--------------------------------

### Install MOSS-TTS-Delay with llama.cpp and ONNX audio

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/llama_cpp/README.md

Installs the package with minimal dependencies for a torch-free setup using ONNX for the audio tokenizer.

```bash
pip install -e ".[llama-cpp-onnx]"
```

--------------------------------

### Setup Environment with uv

Source: https://github.com/openmoss/moss-tts/blob/main/docs/moss_tts_realtime_model_card.md

Configure the project environment using the uv package manager.

```bash
# Install uv first: https://docs.astral.sh/uv/getting-started/installation/
git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install --torch-backend cu128 -e .
cd moss_tts_realtime
```

--------------------------------

### SGLang Backend Quick Start

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Steps to set up and run MOSS-TTS with the SGLang backend for accelerated inference.

```APIDOC
## SGLang Backend (Accelerated Inference)

### Quick Start

1.  **Clone the SGLang repository**
    ```bash
    git clone https://github.com/OpenMOSS/sglang.git
    ```

2.  **Install SGLang**
    ```bash
    pip install -e ./sglang/python[all]
    ```

3.  **(Optional) Fix CuDNN compatibility**
    ```bash
    pip install nvidia-cudnn-cu12==9.16.0.29
    ```

4.  **Download model weights**
    ```bash
    huggingface-cli download OpenMOSS-Team/MOSS-TTS --local-dir weights/MOSS-TTS
    huggingface-cli download OpenMOSS-Team/MOSS-Audio-Tokenizer --local-dir weights/MOSS-Audio-Tokenizer
    ```

5.  **Fuse model and tokenizer weights**
    ```bash
    python scripts/fuse_moss_tts_delay_with_codec.py --model-path weights/MOSS-TTS --codec-model-path weights/MOSS-Audio-Tokenizer --save-path weights/MOSS-TTS-Delay-With-Codec
    ```

6.  **Start the service**
    ```bash
    sglang serve --model-path weights/MOSS-TTS-Delay-With-Codec --delay-pattern --trust-remote-code
    ```

**Note**: The first request might trigger a lengthy compilation step. If the fused output directory exists, use `--overwrite` to replace it.
```

--------------------------------

### Start MOSS-TTS-Realtime Server

Source: https://context7.com/openmoss/moss-tts/llms.txt

Commands to launch the FastAPI server for streaming TTS.

```bash
# Start the FastAPI server
python3 moss_tts_realtime/fast_api.py --host 0.0.0.0 --port 8083

# Or with custom paths
python3 moss_tts_realtime/fast_api.py \
    --model_path OpenMOSS-Team/MOSS-TTS-Realtime \
    --codec_model_path OpenMOSS-Team/MOSS-Audio-Tokenizer \
    --device cuda:0
```

--------------------------------

### Install FlashAttention 2 with uv (Limited RAM)

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Install FlashAttention 2 with MOSS-TTS dependencies using uv, capping build parallelism for machines with limited RAM and many CPU cores.

```bash
MAX_JOBS=4 uv pip install --torch-backend cu128 -e ".[torch-runtime,flash-attn]"
```

--------------------------------

### Install MOSS-TTS-Delay with llama.cpp and TensorRT audio

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/llama_cpp/README.md

Installs the package for maximum performance, utilizing TensorRT for the audio tokenizer.

```bash
pip install -e ".[llama-cpp-trt]"
```

--------------------------------

### Install FlashAttention 2 with uv

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Install FlashAttention 2 with MOSS-TTS dependencies using uv, ensuring PyTorch CUDA wheels are fetched.

```bash
uv pip install --torch-backend cu128 -e ".[torch-runtime,flash-attn]"
```

--------------------------------

### Install SGLang

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Install SGLang and its dependencies using pip. The '[all]' option installs all optional dependencies.

```bash
pip install -e ./sglang/python[all]
```

--------------------------------

### Install Dependencies via Pip

Source: https://github.com/openmoss/moss-tts/blob/main/docs/moss_tts_realtime_model_card.md

Clone the repository and install required packages with CUDA 12.8 support.

```bash
git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e .
cd moss_tts_realtime
```

--------------------------------

### Build llama.cpp

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/llama_cpp/conversion/README.md

Clone the llama.cpp repository and build it to obtain the necessary conversion and quantization tools. This is a one-time setup.

```bash
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j
cd ..
```

--------------------------------

### llama.cpp Backend CLI Usage

Source: https://context7.com/openmoss/moss-tts/llms.txt

Commands for installing, configuring, and running the llama.cpp backend for torch-free inference.

```bash
# Installation (torch-free with ONNX audio)
pip install -e ".[llama-cpp-onnx]"

# Download pre-quantized weights
huggingface-cli download OpenMOSS-Team/MOSS-TTS-GGUF --local-dir weights/MOSS-TTS-GGUF
huggingface-cli download OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX --local-dir weights/MOSS-Audio-Tokenizer-ONNX

# Build the C bridge (requires llama.cpp compiled from source)
cd moss_tts_delay/llama_cpp && bash build_bridge.sh /path/to/llama.cpp && cd ../..

# CLI usage - basic generation
python -m moss_tts_delay.llama_cpp \
    --config configs/llama_cpp/default.yaml \
    --text "Hello, world!" \
    --output output.wav

# With voice cloning
python -m moss_tts_delay.llama_cpp \
    --config configs/llama_cpp/default.yaml \
    --text "Hello!" \
    --reference ref.wav \
    --output output.wav

# Low-memory mode for 8GB GPUs
python -m moss_tts_delay.llama_cpp \
    --config configs/llama_cpp/trt-8gb.yaml \
    --text "Hello!" \
    --output output.wav
```

--------------------------------

### Installation Profiles

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Different installation profiles for MOSS-TTS, each with specific dependencies and use cases.

```APIDOC
## Installation Profiles

### Torch-free (ONNX)

- **Install Command**: `pip install -e ".[llama-cpp-onnx]"`
- **Dependencies**: numpy, onnxruntime-gpu, tokenizers
- **Use Case**: Recommended starting point

### Torch-free (TRT)

- **Install Command**: `pip install -e ".[llama-cpp-trt]"`
- **Dependencies**: numpy, tensorrt, cuda-python
- **Use Case**: Maximum audio tokenizer speed (build engines yourself)

### Torch-accelerated

- **Install Command**: `pip install -e ".[llama-cpp-onnx,llama-cpp-torch]"`
- **Dependencies**: numpy, onnxruntime-gpu, tokenizers, torch
- **Use Case**: GPU-accelerated LM heads (~30x faster)
```

--------------------------------

### Launch MOSS-TTS Training with ZeRO-3 Configuration

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_local/finetuning/README.md

An example of launching the MOSS-TTS training script with specific environment variables set for ZeRO-3 optimization. This includes paths to raw and prepared JSONL files, output directory, and detailed arguments for both data preparation and the SFT trainer.

```bash
RAW_JSONL=train_raw.jsonl \
PREPARED_JSONL=prepared/train_with_codes.jsonl \
OUTPUT_DIR=output/moss_tts_local_sft_zero3 \
ACCELERATE_CONFIG_FILE=moss_tts_local/finetuning/configs/accelerate_zero3_1.7b.yaml \
PREP_ACCELERATE_ARGS_STR='--config_file moss_tts_local/finetuning/configs/accelerate_ddp_8gpu.yaml' \
PREP_EXTRA_ARGS_STR='' \
TRAIN_EXTRA_ARGS_STR='--per-device-batch-size 1 --gradient-accumulation-steps 4 --num-epochs 3 --warmup-ratio 0.03 --mixed-precision bf16 --channelwise-loss-weight 1,32 --gradient-checkpointing' \
bash moss_tts_local/finetuning/run_train.sh
```

--------------------------------

### Install Fine-Tuning Dependencies

Source: https://context7.com/openmoss/moss-tts/llms.txt

Installs necessary packages for fine-tuning, including optional DeepSpeed support.

```bash
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,finetune]"

# For DeepSpeed ZeRO-3 support
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,finetune-deepspeed]"
```

--------------------------------

### All-in-One MOSS-TTS to GGUF Conversion Example

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/llama_cpp/conversion/README.md

A comprehensive script combining all steps: building llama.cpp, extracting weights, converting to f16 GGUF, and quantizing to Q4_K_M. Ensure llama.cpp is built before running.

```bash
# 0. Build llama.cpp (one-time)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release -j && cd ..

# 1. Extract weights
python moss_tts_delay/llama_cpp/conversion/extract_weights.py \
    --model OpenMOSS-Team/MOSS-TTS \
    --output weights/extracted

# 2. Convert to f16 GGUF
python llama.cpp/convert_hf_to_gguf.py \
    weights/extracted/qwen3_backbone \
    --outfile weights/backbone_f16.gguf \
    --outtype f16

# 3. Quantize to Q4_K_M
llama.cpp/build/bin/llama-quantize \
    weights/backbone_f16.gguf \
    weights/backbone_q4km.gguf \
    Q4_K_M

# Done! Use the quantized backbone + embeddings + lm_heads for inference.
# See the llama.cpp backend README for usage instructions.
```

--------------------------------

### Install MOSS-TTS with Torch-accelerated Profile

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Install MOSS-TTS with both ONNX and PyTorch dependencies for GPU-accelerated LM heads.

```bash
pip install -e ".[llama-cpp-onnx,llama-cpp-torch]"
```

--------------------------------

### Install MOSS-TTS Dependencies with uv

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Clone the MOSS-TTS repository, create a virtual environment using uv, and install dependencies with PyTorch CUDA support.

```bash
# Install uv first: https://docs.astral.sh/uv/getting-started/installation/
git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install --torch-backend cu128 -e ".[torch-runtime]"
```

--------------------------------

### Launch MOSS-TTS-Realtime FastAPI Server

Source: https://github.com/openmoss/moss-tts/blob/main/docs/moss_tts_realtime_model_card.md

Starts a FastAPI server for MOSS TTS Realtime. Currently supports a batch size of 1. After starting, you can interact with the server using the provided client script.

```bash
python3 fast_api.py
```

--------------------------------

### Install MOSS-TTS-Delay with PyTorch LM heads acceleration

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/llama_cpp/README.md

Installs the package with TensorRT for audio and PyTorch for accelerated LM heads, offering a balance of performance and features.

```bash
pip install -e ".[llama-cpp-trt,llama-cpp-torch]"
```

--------------------------------

### Start SGLang Service

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Launch the SGLang service with the fused MOSS-TTS model. The '--delay-pattern' flag enables specific inference patterns.

```bash
sglang serve --model-path weights/MOSS-TTS-Delay-With-Codec --delay-pattern --trust-remote-code
```

--------------------------------

### Data Parallel Training (8-GPU)

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/finetuning/README.md

Configure data parallel training for multi-GPU setups using a specified Accelerate configuration file. The training data can be specified using a glob pattern.

```bash
accelerate launch \
    --config_file moss_tts_delay/finetuning/configs/accelerate_ddp_8gpu.yaml \
    moss_tts_delay/finetuning/sft.py \
    --model-path OpenMOSS-Team/MOSS-TTS \
    --train-jsonl 'prepared/train_with_codes.rank*.jsonl' \
    --output-dir output/moss_tts_sft_ddp \
    --per-device-batch-size 1 \
    --gradient-accumulation-steps 4 \
    --mixed-precision bf16 \
    --channelwise-loss-weight 1,32 \
    --gradient-checkpointing
```

--------------------------------

### MOSS-TTS-Realtime: Streaming API Server

Source: https://context7.com/openmoss/moss-tts/llms.txt

This section details the FastAPI server for real-time streaming TTS with session-based dialogue support. It includes instructions on starting the server and a client example for interacting with the API.

```APIDOC
## MOSS-TTS-Realtime: Streaming API Server

MOSS-TTS-Realtime provides a FastAPI server for streaming TTS with session-based multi-turn dialogue support. The server streams PCM audio chunks in real-time as text is pushed.

### Server Setup

```bash
# Start the FastAPI server
python3 moss_tts_realtime/fast_api.py --host 0.0.0.0 --port 8083

# Or with custom paths
python3 moss_tts_realtime/fast_api.py \
    --model_path OpenMOSS-Team/MOSS-TTS-Realtime \
    --codec_model_path OpenMOSS-Team/MOSS-Audio-Tokenizer \
    --device cuda:0
```

### Client Example

```python
# Client example for streaming TTS
import requests
import wave

BASE_URL = "http://localhost:8083"
session_id = "my-session-001"

# 1. Start a new turn
response = requests.post(f"{BASE_URL}/tts/session/start", json={
    "session_id": session_id,
    "user_text": "Hello, how are you?",
    "assistant_text": "",  # Initial text (optional)
    "prompt_audio": "./audio/prompt_audio.mp3",  # Reference audio for voice cloning
    "new_turn": True,
})
print(response.json())

# 2. Push text incrementally (simulating LLM streaming)
for chunk in ["I'm doing ", "great, ", "thank you for asking!"]:
    response = requests.post(f"{BASE_URL}/tts/session/push", json={
        "session_id": session_id,
        "text": chunk,
        "is_final": False,
    })

# 3. Mark turn as complete
response = requests.post(f"{BASE_URL}/tts/session/push", json={
    "session_id": session_id,
    "text": "",
    "is_final": True,
})

# 4. Stream audio response
response = requests.get(f"{BASE_URL}/tts/session/{session_id}/audio", stream=True)
sample_rate = int(response.headers.get("X-Audio-Sample-Rate", 24000))

with wave.open("streaming_output.wav", "wb") as wav_file:
    wav_file.setnchannels(1)
    wav_file.setsampwidth(2)  # 16-bit PCM
    wav_file.setframerate(sample_rate)
    for chunk in response.iter_content(chunk_size=4096):
        if chunk:
            wav_file.writeframes(chunk)

# 5. Close session when done
requests.post(f"{BASE_URL}/tts/session/close", json={"session_id": session_id})
```
```

--------------------------------

### Prepare Data for Single Process

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/finetuning/README.md

Use this script to pre-encode reference and target audio for training. Ensure you have the necessary model and codec paths.

```bash
python moss_tts_delay/finetuning/prepare_data.py \
    --model-path OpenMOSS-Team/MOSS-TTS \
    --codec-path OpenMOSS-Team/MOSS-Audio-Tokenizer \
    --device auto \
    --input-jsonl train_raw.jsonl \
    --output-jsonl train_with_codes.jsonl
```

```bash
python moss_tts_delay/finetuning/prepare_data.py \
    --model-path OpenMOSS-Team/MOSS-TTS \
    --codec-path OpenMOSS-Audio-Tokenizer \
    --device auto \
    --input-jsonl train_raw.jsonl \
    --output-jsonl train_with_codes.jsonl \
    --skip-reference-audio-codes
```

--------------------------------

### Initialize MOSS-VoiceGenerator

Source: https://context7.com/openmoss/moss-tts/llms.txt

Sets up the model for generating voices from text descriptions. Requires normalization of inputs for optimal performance.

```python
from pathlib import Path
import torch
import torchaudio
from transformers import AutoModel, AutoProcessor

processor = AutoProcessor.from_pretrained(
    "OpenMOSS-Team/MOSS-VoiceGenerator",
    trust_remote_code=True,
    normalize_inputs=True,  # Normalize text and instruction input
)
processor.audio_tokenizer = processor.audio_tokenizer.to("cuda")

model = AutoModel.from_pretrained(
    "OpenMOSS-Team/MOSS-VoiceGenerator",
    trust_remote_code=True,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
).to("cuda")
model.eval()
```

--------------------------------

### Install MOSS-TTS Dependencies with Conda/pip

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Clone the MOSS-TTS repository and install its required dependencies, including PyTorch with CUDA support.

```bash
git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"
```

--------------------------------

### Run One-Click Training Launcher

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_realtime/finetuning/README.md

Uses the provided shell script to automate training, optionally with environment variable overrides.

```bash
bash moss_tts_realtime/finetuning/run_train.sh
```

```bash
RAW_JSONL=train_raw.jsonl \
PREPARED_JSONL=prepared/train_with_codes.jsonl \
OUTPUT_DIR=output/moss_tts_realtime_sft_zero3 \
ACCELERATE_CONFIG_FILE=moss_tts_realtime/finetuning/configs/accelerate_zero3_1.7b.yaml \
PREP_ACCELERATE_ARGS_STR='--config_file moss_tts_realtime/finetuning/configs/accelerate_ddp_8gpu.yaml' \
PREP_EXTRA_ARGS_STR='' \
TRAIN_EXTRA_ARGS_STR='--per-device-batch-size 1 --gradient-accumulation-steps 4 --num-epochs 3 --warmup-ratio 0.03 --mixed-precision bf16' \
bash moss_tts_realtime/finetuning/run_train.sh
```

--------------------------------

### Configure Continuation and Prefix Audio

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_local/README.md

Initializes the environment for audio continuation tasks.

```python
import importlib.util
from pathlib import Path
import torch
import torchaudio
from transformers import AutoModel, AutoProcessor, GenerationConfig
# Disable the broken cuDNN SDPA backend
torch.backends.cuda.enable_cudnn_sdp(False)
```

--------------------------------

### Install FlashAttention 2 with Conda/pip

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Install FlashAttention 2 along with MOSS-TTS dependencies for improved speed and reduced GPU memory usage. Ensure your hardware supports it.

```bash
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
```

--------------------------------

### Run data preparation

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_local/finetuning/README.md

Command to pre-encode audio data for training. Use the skip flag to avoid encoding reference audio.

```bash
python moss_tts_local/finetuning/prepare_data.py \
    --model-path OpenMOSS-Team/MOSS-TTS-Local-Transformer \
    --codec-path OpenMOSS-Team/MOSS-Audio-Tokenizer \
    --device auto \
    --input-jsonl train_raw.jsonl \
    --output-jsonl train_with_codes.jsonl
```

```bash
python moss_tts_local/finetuning/prepare_data.py \
    --model-path OpenMOSS-Team/MOSS-TTS-Local-Transformer \
    --codec-path OpenMOSS-Team/MOSS-Audio-Tokenizer \
    --device auto \
    --input-jsonl train_raw.jsonl \
    --output-jsonl train_with_codes.jsonl \
    --skip-reference-audio-codes
```

--------------------------------

### Install FlashAttention 2 with Conda/pip (Limited RAM)

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Install FlashAttention 2 with MOSS-TTS dependencies, capping build parallelism for machines with limited RAM and many CPU cores.

```bash
MAX_JOBS=4 pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
```

--------------------------------

### Initialize Processor

Source: https://github.com/openmoss/moss-tts/blob/main/docs/moss_tts_model_card.md

Loads the AutoProcessor and moves the audio tokenizer to the target device.

```python
processor = AutoProcessor.from_pretrained(
    pretrained_model_name_or_path,
    trust_remote_code=True
)
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
```

--------------------------------

### Prepare Data for Multi-Node/Multi-GPU

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/finetuning/README.md

Launch data preparation across multiple processes using 'accelerate launch'. This splits the dataset into shards for distributed processing.

```bash
accelerate launch --num_processes 16 moss_tts_delay/finetuning/prepare_data.py \
    --model-path OpenMOSS-Team/MOSS-TTS \
    --codec-path OpenMOSS-Team/MOSS-Audio-Tokenizer \
    --device auto \
    --input-jsonl train_raw.jsonl \
    --output-jsonl prepared/train_with_codes.jsonl
```

--------------------------------

### Install DeepSpeed dependencies

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_local/finetuning/README.md

Additional dependencies required for training with DeepSpeed ZeRO-3.

```bash
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,finetune-deepspeed]"
```

--------------------------------

### Initialize MOSS-VoiceGenerator Model and Processor

Source: https://github.com/openmoss/moss-tts/blob/main/docs/moss_voice_generator_model_card.md

Sets up the environment for MOSS-VoiceGenerator by initializing the model, processor, and determining the optimal attention implementation based on hardware and software availability. Handles CUDA and CPU fallbacks.

```python
from pathlib import Path
import importlib.util
import torch
import torchaudio
from transformers import AutoModel, AutoProcessor
# Disable the broken cuDNN SDPA backend
torch.backends.cuda.enable_cudnn_sdp(False)
# Keep these enabled as fallbacks
torch.backends.cuda.enable_flash_sdp(True)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_math_sdp(True)

pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-VoiceGenerator"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

def resolve_attn_implementation() -> str:
    # Prefer FlashAttention 2 when package + device conditions are met.
    if (
        device == "cuda"
        and importlib.util.find_spec("flash_attn") is not None
        and dtype in {torch.float16, torch.bfloat16}
    ):
        major, _ = torch.cuda.get_device_capability()
        if major >= 8:
            return "flash_attention_2"

    # CUDA fallback: use PyTorch SDPA kernels.
    if device == "cuda":
        return "sdpa"

    # CPU fallback.
    return "eager"


attn_implementation = resolve_attn_implementation()
print(f"[INFO] Using attn_implementation={attn_implementation}")

processor = AutoProcessor.from_pretrained(
    pretrained_model_name_or_path,
    trust_remote_code=True,
    normalize_inputs=True, # normalize text and instruction input
)
processor.audio_tokenizer = processor.audio_tokenizer.to(device)

```

--------------------------------

### Client Streaming API for MOSS-TTS

Source: https://context7.com/openmoss/moss-tts/llms.txt

Example of interacting with the MOSS-TTS-Realtime session-based API.

```python
# Client example for streaming TTS
import requests
import wave

BASE_URL = "http://localhost:8083"
session_id = "my-session-001"

# 1. Start a new turn
response = requests.post(f"{BASE_URL}/tts/session/start", json={
    "session_id": session_id,
    "user_text": "Hello, how are you?",
    "assistant_text": "",  # Initial text (optional)
    "prompt_audio": "./audio/prompt_audio.mp3",  # Reference audio for voice cloning
    "new_turn": True,
})
print(response.json())

# 2. Push text incrementally (simulating LLM streaming)
for chunk in ["I'm doing ", "great, ", "thank you for asking!"]:
    response = requests.post(f"{BASE_URL}/tts/session/push", json={
        "session_id": session_id,
        "text": chunk,
        "is_final": False,
    })

# 3. Mark turn as complete
response = requests.post(f"{BASE_URL}/tts/session/push", json={
    "session_id": session_id,
    "text": "",
    "is_final": True,
})

# 4. Stream audio response
response = requests.get(f"{BASE_URL}/tts/session/{session_id}/audio", stream=True)
sample_rate = int(response.headers.get("X-Audio-Sample-Rate", 24000))

with wave.open("streaming_output.wav", "wb") as wav_file:
    wav_file.setnchannels(1)
    wav_file.setsampwidth(2)  # 16-bit PCM
    wav_file.setframerate(sample_rate)
    for chunk in response.iter_content(chunk_size=4096):
        if chunk:
            wav_file.writeframes(chunk)

# 5. Close session when done
requests.post(f"{BASE_URL}/tts/session/close", json={"session_id": session_id})
```

--------------------------------

### Run Fine-Tuning Training

Source: https://context7.com/openmoss/moss-tts/llms.txt

Executes training using various strategies like single-GPU, DDP, or FSDP.

```bash
accelerate launch moss_tts_delay/finetuning/sft.py \
    --model-path OpenMOSS-Team/MOSS-TTS \
    --train-jsonl train_with_codes.jsonl \
    --output-dir output/moss_tts_sft \
    --per-device-batch-size 1 \
    --gradient-accumulation-steps 8 \
    --learning-rate 1e-5 \
    --warmup-ratio 0.03 \
    --num-epochs 3 \
    --mixed-precision bf16 \
    --channelwise-loss-weight 1,32 \
    --gradient-checkpointing
```

```bash
accelerate launch \
    --config_file moss_tts_delay/finetuning/configs/accelerate_ddp_8gpu.yaml \
    moss_tts_delay/finetuning/sft.py \
    --model-path OpenMOSS-Team/MOSS-TTS \
    --train-jsonl 'prepared/train_with_codes.rank*.jsonl' \
    --output-dir output/moss_tts_sft_ddp \
    --per-device-batch-size 1 \
    --gradient-accumulation-steps 4 \
    --mixed-precision bf16 \
    --gradient-checkpointing
```

```bash
accelerate launch \
    --config_file moss_tts_delay/finetuning/configs/accelerate_fsdp_8b.yaml \
    moss_tts_delay/finetuning/sft.py \
    --model-path OpenMOSS-Team/MOSS-TTS \
    --train-jsonl 'prepared/train_with_codes.rank*.jsonl' \
    --output-dir output/moss_tts_sft_fsdp \
    --per-device-batch-size 1 \
    --gradient-accumulation-steps 4 \
    --mixed-precision bf16 \
    --gradient-checkpointing
```

--------------------------------

### Fine-Tuning Data Formats

Source: https://context7.com/openmoss/moss-tts/llms.txt

Examples of JSONL data formats for different MOSS-TTS training tasks.

```jsonl
{"audio":"./data/utt0001.wav","text":"Actually, I noticed that I am very sensitive to other people's emotions.","language":"en"}
{"audio":"./data/utt0002.wav","text":"She said she would be here by noon.","ref_audio":"./data/ref.wav","language":"en"}

# MOSS-TTSD format (multi-speaker dialogue)
{"audio":"./data/dialog.wav","text":"[S1] Hello there. [S2] Hi, how are you?","reference":["./data/s1_ref.wav", null],"language":"en"}

# MOSS-SoundEffect format
{"audio":"./data/rain.wav","ambient_sound":"Rolling thunder with steady rainfall."}

# MOSS-VoiceGenerator format
{"audio":"./data/old_man.wav","text":"My old back is giving me trouble.","instruction":"A tired, hoarse elderly voice."}
```

--------------------------------

### Initialize MOSS-TTS-Realtime Inference

Source: https://context7.com/openmoss/moss-tts/llms.txt

Sets up the model, tokenizer, and codec for non-streaming real-time voice inference.

```python
import torch
import torchaudio
from transformers import AutoTokenizer, AutoModel
from mossttsrealtime.modeling_mossttsrealtime import MossTTSRealtime
from inferencer import MossTTSRealtimeInference

CODEC_SAMPLE_RATE = 24000
device = "cuda"

model = MossTTSRealtime.from_pretrained(
    "OpenMOSS-Team/MOSS-TTS-Realtime",
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
).to(device)

tokenizer = AutoTokenizer.from_pretrained("OpenMOSS-Team/MOSS-TTS-Realtime")
codec = AutoModel.from_pretrained("OpenMOSS-Team/MOSS-Audio-Tokenizer", trust_remote_code=True).eval().to(device)

inferencer = MossTTSRealtimeInference(
    model,
    tokenizer,
    max_length=5000,
    codec=codec,
    codec_sample_rate=CODEC_SAMPLE_RATE,
    codec_encode_kwargs={"chunk_duration": 8},
)

# Multiple turns of dialogue
text = [
    "Welcome to the world of MOSS TTS Realtime.",
    "Experience how text transforms into smooth, human-like speech in real time.",
]
```

--------------------------------

### Launch Gradio Streaming Demo

Source: https://github.com/openmoss/moss-tts/blob/main/docs/moss_tts_realtime_model_card.md

Launches the interactive Gradio demo for MOSS TTS Realtime, which supports streaming output. It's recommended to use SDPA + torch.compile for faster inference.

```bash
python3 app.py
```

--------------------------------

### Initialize MOSS-TTS and Generate Speech

Source: https://github.com/openmoss/moss-tts/blob/main/docs/moss_tts_model_card.md

Configures the environment, resolves attention implementation, and prepares the processor for text-to-speech generation.

```python
from pathlib import Path
import importlib.util
import torch
import torchaudio
from transformers import AutoModel, AutoProcessor
# Disable the broken cuDNN SDPA backend
torch.backends.cuda.enable_cudnn_sdp(False)
# Keep these enabled as fallbacks
torch.backends.cuda.enable_flash_sdp(True)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_math_sdp(True)


pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

def resolve_attn_implementation() -> str:
    # Prefer FlashAttention 2 when package + device conditions are met.
    if (
        device == "cuda"
        and importlib.util.find_spec("flash_attn") is not None
        and dtype in {torch.float16, torch.bfloat16}
    ):
        major, _ = torch.cuda.get_device_capability()
        if major >= 8:
            return "flash_attention_2"

    # CUDA fallback: use PyTorch SDPA kernels.
    if device == "cuda":
        return "sdpa"

    # CPU fallback.
    return "eager"


attn_implementation = resolve_attn_implementation()
print(f"[INFO] Using attn_implementation={attn_implementation}")

processor = AutoProcessor.from_pretrained(
    pretrained_model_name_or_path,
    trust_remote_code=True,
)
processor.audio_tokenizer = processor.audio_tokenizer.to(device)

text_1 = "亲爱的你，\n你好呀。\n\n今天，我想用最认真、最温柔的声音，对你说一些重要的话。\n这些话，像一颗小小的星星，希望能在你的心里慢慢发光。\n\n首先，我想祝你——\n每天都能平平安安、快快乐乐。\n\n希望你早上醒来的时候，\n窗外有光，屋子里很安静，\n你的心是轻轻的，没有着急，也没有害怕。\n\n希望你吃饭的时候胃口很好，\n走路的时候脚步稳稳，\n晚上睡觉的时候，能做一个又一个甜甜的梦。\n\n我希望你能一直保持好奇心。\n对世界充满问题，\n对天空、星星、花草、书本和故事感兴趣。\n当你问“为什么”的时候，\n希望总有人愿意认真地听你说话。\n\n我也希望你学会温柔。\n温柔地对待朋友，\n温柔地对待小动物，\n也温柔地对待自己。\n\n如果有一天你犯了错，\n请不要太快责怪自己，\n因为每一个认真成长的人，\n都会在路上慢慢学会更好的方法。\n\n愿你拥有勇气。\n当你站在陌生的地方时，\n当你第一次举手发言时，\n当你遇到困难、感到害怕的时候，\n希望你能轻轻地告诉自己：\n“我可以试一试。”\n\n就算没有一次成功，也没有关系。\n失败不是坏事，\n它只是告诉你，你正在努力。\n\n我希望你学会分享快乐。\n把开心的事情告诉别人，\n把笑声送给身边的人，\n因为快乐被分享的时候，\n会变得更大、更亮。\n\n如果有一天你感到难过，\n我希望你知道——\n难过并不丢脸，\n哭泣也不是软弱。\n\n愿你能找到一个安全的地方，\n慢慢把心里的话说出来，\n然后再一次抬起头，看见希望。\n\n我还希望你能拥有梦想。\n这个梦想也许很大，\n也许很小，\n也许现在还说不清楚。\n\n没关系。\n梦想会和你一起长大，\n在时间里慢慢变得清楚。\n\n最后，我想送你一个最最重要的祝福：\n\n愿你被世界温柔对待，\n也愿你成为一个温柔的人。\n\n愿你的每一天，\n都值得被记住，\n都值得被珍惜。\n\n亲爱的你，\n请记住，\n你是独一无二的，\n你已经很棒了，\n而你的未来，\n一定会慢慢变得闪闪发光。\n\n祝你健康、勇敢、幸福，\n祝你永远带着笑容向前走。"
text_2 = "We stand on the threshold of the AI era.\nArtificial intelligence is no longer just a concept in laboratories, but is entering every industry, every creative endeavor, and every decision. It has learned to see, hear, speak, and think, and is beginning to become an extension of human capabilities. AI is not about replacing humans, but about amplifying human creativity, making knowledge more equitable, more efficient, and allowing imagination to reach further. A new era, jointly shaped by humans and intelligent systems, has arrived."
text_3 = "nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？"
text_4 = "nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？"
text_5 = "您好，请问您来自哪 zuo4 cheng2 shi4？"
text_6 = "/həloʊ, meɪ aɪ æsk wɪtʃ sɪti juː ɑːr frʌm?/"
```

--------------------------------

### Launch Training with One-Click Script

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/finetuning/README.md

Executes the training pipeline using the provided shell script. Supports environment variable overrides for data paths, output directories, and training arguments.

```bash
bash moss_tts_delay/finetuning/run_train.sh
```

```bash
RAW_JSONL=train_raw.jsonl \
PREPARED_JSONL=prepared/train_with_codes.jsonl \
OUTPUT_DIR=output/moss_tts_sft_zero3 \
ACCELERATE_CONFIG_FILE=moss_tts_delay/finetuning/configs/accelerate_zero3_8b.yaml \
PREP_ACCELERATE_ARGS_STR='--config_file moss_tts_delay/finetuning/configs/accelerate_ddp_8gpu.yaml' \
PREP_EXTRA_ARGS_STR='' \
TRAIN_EXTRA_ARGS_STR='--per-device-batch-size 1 --gradient-accumulation-steps 4 --num-epochs 3 --warmup-ratio 0.03 --mixed-precision bf16 --channelwise-loss-weight 1,32 --gradient-checkpointing' \
bash moss_tts_delay/finetuning/run_train.sh
```

--------------------------------

### SGLang Backend Request and Response

Source: https://github.com/openmoss/moss-tts/blob/main/README.md

Example of how to make a request to the MOSS-TTS SGLang backend and the expected response format.

```APIDOC
## Request and Response

### MOSS-TTS (Delay) Request Example

```bash
curl -X POST http://localhost:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Added SGLang backend support for efficient inference.",
    "audio_data": "https://cdn.jsdelivr.net/gh/OpenMOSS/MOSS-TTSD@main/legacy/v0.7/examples/zh_spk1_moon.wav",
    "sampling_params": {
      "max_new_tokens": 512,
      "temperature": 1.7,
      "top_p": 0.8,
      "top_k": 25
    }
  }'
```

### Request Parameters

- **text** (string) - Required - The text content to be synthesized. Can prepend `${token:25}` for token control.
- **audio_data** (string) - Optional - Reference audio. Can be a `<path-to-audio-file>` or `data:audio/wav;base64,{b64_audio}`. If omitted, a random timbre is used.
- **sampling_params** (object) - Optional - Parameters for sampling.
  - **max_new_tokens** (integer) - Maximum number of new tokens to generate.
  - **temperature** (float) - Controls randomness. Higher values increase randomness.
  - **top_p** (float) - Nucleus sampling parameter. Considers tokens cumulatively until their total probability mass exceeds `top_p`.
  - **top_k** (integer) - Top-k sampling parameter. Considers only the `top_k` most likely tokens.
```

--------------------------------

### Download model weights and audio codecs

Source: https://github.com/openmoss/moss-tts/blob/main/moss_tts_delay/llama_cpp/README_zh.md

Use huggingface-cli to download the required GGUF backbone and ONNX audio models.

```bash
huggingface-cli download OpenMOSS-Team/MOSS-TTS-GGUF --local-dir weights/MOSS-TTS-GGUF
```

```bash
huggingface-cli download OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX --local-dir weights/MOSS-Audio-Tokenizer-ONNX
```

--------------------------------

### Define Input Prompts for Speech Generation

Source: https://github.com/openmoss/moss-tts/blob/main/docs/moss_ttsd_model_card.md

Sets up the input text and reference audio prompts for generating speech. This includes URLs for prompt audio and corresponding text segments for different speakers.

```python
# Use audio from ./assets/audio to avoid downloading from the cloud.
prompt_audio_speaker1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_02_s1.wav"
prompt_audio_speaker2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_02_s2.wav"
prompt_text_speaker1 = "[S1] In short, we embarked on a mission to make America great again for all Americans."
prompt_text_speaker2 = "[S2] NVIDIA reinvented computing for the first time after 60 years. In fact, Erwin at IBM knows quite well that the computer has largely been the same since the 60s."

text_to_generate = "[S1] Listen, let's talk business. China. I'm hearing things. People are saying they're catching up. Fast. What's the real scoop? Their AI—is it a threat? [S2] Well, the pace of innovation there is extraordinary, honestly. They have the researchers, and they have the drive. [S1] Extraordinary? I don't like that. I want us to be extraordinary. Are they winning? [S2] I wouldn't say winning, but their progress is very promising. They are building massive clusters. They're very determined. [S1] Promising. There it is. I hate that word. When China is promising, it means we're losing. It's a disaster, Jensen. A total disaster. "
```