### Setup and Run API with Docker (Bash & Python)

Source: https://github.com/mtkresearch/breezyvoice/blob/main/README.md

Sets up and runs the BreezyVoice API service using Docker Compose. After the container is running, it demonstrates how to install the OpenAI Python client and run the `openai_api_inference.py` script to interact with the API.

```bash
$ docker compose up -d --build
# after the container is up
$ pip install openai
$ python openai_api_inference.py
```

--------------------------------

### Run Matcha-TTS Training Scripts

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/README.md

These commands initiate the training process for the Matcha-TTS model. They cover standard training, minimum memory configuration, and multi-GPU training setups.

```bash
make train-ljspeech
```

```bash
python matcha/train.py experiment=ljspeech
```

```bash
python matcha/train.py experiment=ljspeech_min_memory
```

```bash
python matcha/train.py experiment=ljspeech trainer.devices=[0,1]
```

--------------------------------

### Install ONNX Runtime for Export and Inference

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/README.md

These commands install the necessary libraries for exporting Matcha checkpoints to ONNX format and for running inference using ONNX Runtime. GPU support is also included.

```bash
pip install onnx
```

```bash
pip install onnxruntime
```

```bash
pip install onnxruntime-gpu  # for GPU inference
```

--------------------------------

### API Endpoints - BreezyVoice

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Examples of interacting with the BreezyVoice REST API using curl. Shows how to list available models via GET request and generate speech via POST request with JSON payload. The API is served on http://localhost:8080.

```bash
curl http://localhost:8080/v1/models
```

```bash
curl -X POST http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "今天天氣真好",
    "response_format": "wav",
    "speed": 1.0
  }' \
  --output speech.wav
```

--------------------------------

### REST API Server and Client - BreezyVoice

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

FastAPI-based REST API implementing OpenAI's TTS specification for speech synthesis. Includes Python code to start the server using uvicorn and a client example using the OpenAI SDK to generate and save speech. The API is accessible at http://localhost:8080.

```python
import uvicorn
uvicorn.run("api:app", host="0.0.0.0", port=8080)
```

```python
from pathlib import Path
import openai

# Initialize client
client = openai.Client(
    base_url="http://localhost:8080",
    api_key="sk-template"
)

# Generate speech
speech_file_path = Path(__file__).parent / "./results/speech.wav"
response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="冷氣團南下 北部轉涼白天氣溫降8度"
)

# Save audio
with open(speech_file_path, "wb") as audio_file:
    audio_file.write(response.content)
```

--------------------------------

### Project Configuration and Metadata

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/matcha_tts.egg-info/SOURCES.txt

Manages project dependencies, build configurations, and package metadata. 'pyproject.toml' is used for modern Python packaging standards, while 'setup.py' provides traditional setuptools integration. 'requirements.txt' lists runtime dependencies.

```toml
[tool.poetry]
name = "matcha-tts"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"

```

```python
from setuptools import find_packages, setup

setup(
    name="matcha-tts",
    version="0.1.0",
    packages=find_packages(exclude=["tests", "scripts", "docs"]),
    install_requires=[
        "torch>=1.12.0",
        "torchmetrics>=0.9.3",
        "numpy>=1.21.2",
        "scipy>=1.7.1",
        "librosa>=0.9.2",
        "soundfile>=0.11.0",
        "datasets>=2.0.0",
        "pandas>=1.3.4",
        "tensorboard>=2.9.0",
        "transformers>=4.19.0",
        "huggingface-hub>=0.7.0",
        "sentencepiece>=0.1.97",
        "rich>=11.0.0",
        "protobuf>=3.20.0",
        "tqdm>=4.64.0",
        "accelerate>=0.12.0",
        "einops>=0.4.1",
        "pyyaml>=6.0",
        "hydra-core>=1.2.0",
        "omegaconf>=2.2.0",
        "wandb>=0.13.1",
        "tensorly>=0.7.0",
        "x-clip>=0.0.5",
        "einops-jax>=0.2.0",
        "jax>=0.3.25",
        "jaxlib>=0.3.25",
        "optax>=0.1.0",
        "flax>=0.6.3",
        "nbformat>=5.3.0",
        "gradio>=3.0.0",
        "nltk>=3.7",
        "ftfy>=6.1.1",
        "rouge-score>=0.1.2",
        "seqeval>=1.2.2",
        "datasets[audio]>=2.0.0",
        "torchaudio>=0.12.0",
        "ffmpeg-python>=0.2.0",
        "pydub>=0.25.1"
    ],
    extras_require={
        "dev": [
            "black",
            "isort",
            "flake8",
            "mypy",
            "pre-commit",
            "pytest",
            "pytest-cov",
            "pytest-mock",
            "pytest-timeout",
            "torchvision"
        ]
    },
    python_requires=">=3.9"
)

```

```text
torch>=1.12.0
torchmetrics>=0.9.3
numpy>=1.21.2
scipy>=1.7.1
librosa>=0.9.2
soundfile>=0.11.0
datasets>=2.0.0
pandas>=1.3.4
tensorboard>=2.9.0
transformers>=4.19.0
huggingface-hub>=0.7.0
sentencepiece>=0.1.97
rich>=11.0.0
protobuf>=3.20.0
tqdm>=4.64.0
accelerate>=0.12.0
einops>=0.4.1
pyyaml>=6.0
hydra-core>=1.2.0
omegaconf>=2.2.0
wandb>=0.13.1
tensorly>=0.7.0
x-clip>=0.0.5
einops-jax>=0.2.0
jax>=0.3.25
jaxlib>=0.3.25
optax>=0.1.0
flax>=0.6.3
nbformat>=5.3.0
gradio>=3.0.0
nltk>=3.7
ftfy>=6.1.1
rouge-score>=0.1.2
seqeval>=1.2.2
datasets[audio]>=2.0.0
torchaudio>=0.12.0
ffmpeg-python>=0.2.0
pydub>=0.25.1

```

--------------------------------

### Docker Deployment and GPU Configuration - BreezyVoice

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Instructions for deploying BreezyVoice using Docker Compose, including building the service and enabling GPU support. The compose file specifies the build context, port mapping, Hugging Face cache volume, and NVIDIA GPU resource reservation.

```bash
# Build and start the service
docker compose up -d --build

# Service is now available at http://localhost:8080
# Use with OpenAI client
pip install openai
python openai_api_inference.py
```

```yaml
services:
  app:
    build: .
    ports:
      - "8080:8080"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

--------------------------------

### Import Necessary Libraries for Matcha-TTS and HiFiGAN

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

Imports core libraries for deep learning, audio processing, and TTS model implementation. Includes PyTorch for tensor operations, soundfile for audio I/O, tqdm for progress bars, and specific modules from the matcha and hifigan libraries for model architecture and utilities.

```python
import datetime as dt
from pathlib import Path

import IPython.display as ipd
import numpy as np
import soundfile as sf
import torch
from tqdm.auto import tqdm

# Hifigan imports
from matcha.hifigan.config import v1
from matcha.hifigan.denoiser import Denoiser
from matcha.hifigan.env import AttrDict
from matcha.hifigan.models import Generator as HiFiGAN
# Matcha imports
from matcha.models.matcha_tts import MatchaTTS
from matcha.text import sequence_to_text, text_to_sequence
from matcha.utils.model import denormalize
from matcha.utils.utils import get_user_data_dir, intersperse
```

--------------------------------

### HiFiGAN Vocoder Configuration

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/matcha_tts.egg-info/SOURCES.txt

Defines the configuration parameters for the HiFiGAN vocoder, a neural network used for synthesizing high-fidelity audio waveforms from mel-spectrograms. This file specifies model architecture and training settings.

```python
from dataclasses import dataclass, field

@dataclass
class HiFiGANConfig:
    model_name: str = "hifigan"
    upsample_kernels: list = field(default_factory=lambda: [16, 16, 2, 2, 2, 2, 2, 2])
    upsample_initial_channels: int = 512
    upsample_res_channels: int = 512
    upsample_res_num: int = 3
    num_mels: int = 80
    num_classes: int = 1 # For multi-speaker, change to num_speakers
    output_wav_channels: int = 1
    data_path: str = ""
    batch_size: int = 16
    learning_rate: float = 0.0002
    epochs: int = 1000
    save_interval: int = 50
    log_interval: int = 50
    device: str = "cuda"

```

--------------------------------

### Audio Processing Utilities

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/matcha_tts.egg-info/SOURCES.txt

Contains utility functions for audio processing, including loading, saving, and manipulating audio signals. These functions are essential for preparing training data and post-processing generated audio.

```python
import librosa
import soundfile as sf
import numpy as np

def load_audio(path, sr=22050):
    """Loads an audio file."""
    audio, _ = librosa.load(path, sr=sr)
    return audio

def save_audio(audio, path, sr=22050):
    """Saves an audio array to a file."""
    sf.write(path, audio, sr)

def normalize_audio(audio):
    """Normalizes audio to [-1, 1]."""
    return audio / np.max(np.abs(audio))

```

--------------------------------

### Core Text-to-Speech Model Implementation

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/matcha_tts.egg-info/SOURCES.txt

Implements the main Matcha TTS model, including base classes, attention mechanisms, and text encoding. This module defines the architecture for generating speech from text.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F

from matcha.models.components.text_encoder import TextEncoder
from matcha.models.components.decoder import Decoder
from matcha.models.components.flow_matching import FlowMatching

class MatchaTTS(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.text_encoder = TextEncoder(config.text_encoder)
        self.decoder = Decoder(config.decoder)
        self.flow_matching = FlowMatching(config.flow_matching)

    def forward(self, text_input, mel_input=None, spk_embed=None):
        # Text encoding
        encoded_text = self.text_encoder(text_input)

        # Flow matching (if training)
        if mel_input is not None:
            loss_fm = self.flow_matching(
                encoded_text,
                mel_input,
                spk_embed=spk_embed
            )
        else:
            loss_fm = None

        # Decoder
        mel_output = self.decoder(
            encoded_text,
            spk_embed=spk_embed,
            reverse_conditioning=mel_input is None
        )

        return mel_output, loss_fm

```

--------------------------------

### Load HiFiGAN Vocoder and Denoiser

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

Loads the HiFiGAN vocoder model and initializes a denoiser. The function configures the vocoder using predefined settings (v1), loads weights from a checkpoint, and prepares it for inference. The denoiser is used to clean up the synthesized audio.

```python
def load_vocoder(checkpoint_path):
    h = AttrDict(v1)
    hifigan = HiFiGAN(h).to(device)
    hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device)['generator'])
    _ = hifigan.eval()
    hifigan.remove_weight_norm()
    return hifigan

vocoder = load_vocoder(HIFIGAN_CHECKPOINT)
denoiser = Denoiser(vocoder, mode='zeros')
```

--------------------------------

### Docker Deployment

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Deploy BreezyVoice as a containerized service with GPU support using Docker Compose.

```APIDOC
## Docker Deployment

### Description
Deploy BreezyVoice as a containerized service with GPU support using Docker Compose.

### Method
Docker Compose

### Parameters
None

### Commands
```bash
# Build and start the service
docker compose up -d --build
```

### Accessing the Service
The service will be available at `http://localhost:8080` after starting.

### Docker Compose Configuration (`compose.yaml`)
```yaml
services:
  app:
    build: .
    ports:
      - "8080:8080"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```
```

--------------------------------

### Run Single Inference Script (Python)

Source: https://github.com/mtkresearch/breezyvoice/blob/main/README.md

Executes the `single_inference.py` script for text-to-speech conversion. Requires text and an optional reference audio path. The script synthesizes speech from the provided text using a specified speaker's style.

```python
python single_inference.py --text_to_speech [text to be converted into audio] --audio_path [reference audio file]
python single_inference.py --content_to_synthesize "今天天氣真好[:ㄏㄠ3]" --speaker_prompt_audio_path "./data/example.wav"
```

--------------------------------

### Run ONNX Inference with Matcha Model

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/README.md

This command performs inference using an exported ONNX model. It takes the ONNX model path, input text, and an output directory. Synthesis parameters like temperature and speaking rate can also be controlled.

```bash
python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs
```

```bash
python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --temperature 0.4 --speaking_rate 0.9 --spk 0
```

```bash
python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --gpu
```

```bash
python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --vocoder hifigan.small.onnx
```

--------------------------------

### Cython/C Extension for Monotonic Alignment

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/matcha_tts.egg-info/SOURCES.txt

Provides a Cython/C implementation for efficient monotonic alignment, crucial for tasks like attention alignment in sequence-to-sequence models. This speeds up critical computation parts.

```python
# matcha/utils/monotonic_align/core.pyx
cdef extern from "core.h":
    void fused_gcd(int* x, int* y, int* x_out, int* y_out, int length, int step)

cdef public np.ndarray fused_gcd_numpy(np.ndarray x, np.ndarray y):
    cdef int length = x.shape[0]
    cdef int step = 1
    cdef np.ndarray x_out = np.zeros_like(x)
    cdef np.ndarray y_out = np.zeros_like(y)
    fused_gcd(<int*>x.data, <int*>y.data, <int*>x_out.data, <int*>y_out.data, length, step)
    return x_out, y_out

```

```c
// matcha/utils/monotonic_align/core.c
#include <math.h>

void fused_gcd(int* x, int* y, int* x_out, int* y_out, int length, int step) {
    for (int i = 0; i < length; ++i) {
        x_out[i] = x[i];
        y_out[i] = y[i];
    }
    // Placeholder for actual GCD logic if needed, or this might be a stub for Cython integration
}

```

--------------------------------

### Define Checkpoint and Output Paths

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

Specifies the file paths for the Matcha-TTS checkpoint, the HiFiGAN vocoder checkpoint, and the directory for saving synthesized audio outputs. Uses Path objects for cross-platform compatibility.

```python
MATCHA_CHECKPOINT = get_user_data_dir()/'matcha_ljspeech.ckpt'
HIFIGAN_CHECKPOINT = get_user_data_dir() / "hifigan_T2_v1"
OUTPUT_FOLDER = "synth_output"
```

--------------------------------

### FastAPI Server Configuration with Pydantic Settings

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Configures a FastAPI server using Pydantic's BaseSettings for environment-based configuration. It defines settings such as API key, model path, and speaker prompt audio path, which can be set via environment variables.

```python
from pydantic_settings import BaseSettings
from pydantic import Field

# Configure via environment variables or defaults
class Settings(BaseSettings):
    api_key: str = Field(
        default="",
        description="Specifies the API key used to authenticate the user."
    )
    model_path: str = Field(
        default="MediaTek-Research/BreezyVoice",
        description="Specifies the model used for speech synthesis."
    )
    speaker_prompt_audio_path: str = Field(
        default="./data/example.wav",
        description="Specifies the path to the prompt speech audio file."
    )
    speaker_prompt_text_transcription: str = Field(
        default="在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。",
        description="Specifies the transcription of the speaker prompt audio."
    )

# Set via environment variables
# export MODEL_PATH=/path/to/custom/model
# export API_KEY=your-secret-key
# export SPEAKER_PROMPT_AUDIO_PATH=/path/to/default/speaker.wav
# python api.py
```

--------------------------------

### Text Cleaning and Preprocessing

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/matcha_tts.egg-info/SOURCES.txt

Implements functions for cleaning and preprocessing text data for text-to-speech models. This includes normalizing numbers, expanding abbreviations, and handling symbols.

```python
import re
import unicodedata

def expand_numbers(text):
    """Expands numbers in the text."""
    # Implementation details for number expansion
    pass

def normalize_text(text):
    """Normalizes Unicode characters in the text."""
    return unicodedata.normalize('NFKC', text)

def text_cleaners(text):
    """Applies a sequence of text cleaning functions."""
    text = normalize_text(text)
    text = expand_numbers(text)
    # Add more cleaning steps as needed
    return text

```

--------------------------------

### Python Audio Loading and Resampling Utilities

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Provides utility functions for loading audio files and resampling them to specific sample rates, essential for model input and output handling.

```python
from cosyvoice.utils.file_utils import load_wav
import torchaudio
import torch

# Load and resample audio to 16kHz (required for model input)
audio_16k = load_wav("input_audio.wav", target_sr=16000)
print(f"Audio shape: {audio_16k.shape}")  # torch.Tensor [1, samples]

# Load audio at 22.05kHz (model output sample rate)
audio_22k = load_wav("input_audio.wav", target_sr=22050)

# Manual resample example
audio, sr = torchaudio.load("input_audio.wav")
resampler = torchaudio.transforms.Resample(orig_freq=sr, new_freq=16000)
audio_16k = resampler(audio)
```

--------------------------------

### Export Matcha Checkpoint to ONNX Format

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/README.md

This command exports a trained Matcha checkpoint to an ONNX file. The `--n-timesteps` argument specifies the number of time steps, and optional vocoder arguments can embed the vocoder into the exported graph.

```bash
python3 -m matcha.onnx.export matcha.ckpt model.onnx --n-timesteps 5
```

--------------------------------

### Define Text for Synthesis

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

Sets up a list of strings, where each string is a piece of text to be converted into speech. This is the input data for the synthesis process.

```python
texts = [
    "The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent."
]
```

--------------------------------

### REST API /v1/models

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

List available speech synthesis models supported by the API.

```APIDOC
## REST API

### GET /v1/models

#### Description
List available speech synthesis models.

#### Method
GET

#### Endpoint
`/v1/models`

#### Parameters
##### Query Parameters
None

#### Request Example
```bash
curl http://localhost:8080/v1/models
```

#### Response
##### Success Response (200)
(Response structure not provided in source text, assumed to be a list of model objects)

#### Response Example
(Example not provided in source text)
```

--------------------------------

### Load Matcha-TTS Model from Checkpoint

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

Defines a function to load a pre-trained Matcha-TTS model from a given checkpoint file and set it to evaluation mode. It also includes a helper to count model parameters.

```python
def load_model(checkpoint_path):
    model = MatchaTTS.load_from_checkpoint(checkpoint_path, map_location=device)
    model.eval()
    return model
count_params = lambda x: f"{sum(p.numel() for p in x.parameters()):,}"


model = load_model(MATCHA_CHECKPOINT)
print(f"Model loaded! Parameter count: {count_params(model)}")
```

--------------------------------

### Synthesize Speech Audio Utilities

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

Provides helper functions for text processing, speech synthesis, converting mel-spectrograms to waveforms, and saving generated audio. `process_text` converts input text to sequences, `synthesise` generates mel-spectrograms, `to_waveform` converts mel to audio using the vocoder, and `save_to_folder` saves audio files.

```python
@torch.inference_mode()
def process_text(text: str):
    x = torch.tensor(intersperse(text_to_sequence(text, ['english_cleaners2']), 0),dtype=torch.long, device=device)[None]
    x_lengths = torch.tensor([x.shape[-1]],dtype=torch.long, device=device)
    x_phones = sequence_to_text(x.squeeze(0).tolist())
    return {
        'x_orig': text,
        'x': x,
        'x_lengths': x_lengths,
        'x_phones': x_phones
    }


@torch.inference_mode()
def synthesise(text, spks=None):
    text_processed = process_text(text)
    start_t = dt.datetime.now()
    output = model.synthesise(
        text_processed['x'], 
        text_processed['x_lengths'],
        n_timesteps=n_timesteps,
        temperature=temperature,
        spks=spks,
        length_scale=length_scale
    )
    # merge everything to one dict    
    output.update({'start_t': start_t, **text_processed})
    return output

@torch.inference_mode()
def to_waveform(mel, vocoder):
    audio = vocoder(mel).clamp(-1, 1)
    audio = denoiser(audio.squeeze(0), strength=0.00025).cpu().squeeze()
    return audio.cpu().squeeze()
    
def save_to_folder(filename: str, output: dict, folder: str):
    folder = Path(folder)
    folder.mkdir(exist_ok=True, parents=True)
    np.save(folder / f'{filename}', output['mel'].cpu().numpy())
    sf.write(folder / f'{filename}.wav', output['waveform'], 22050, 'PCM_24')
```

--------------------------------

### Configure Synthesis Parameters

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

Sets key parameters for the speech synthesis process: `n_timesteps` controls the number of steps in the ODE solver for synthesis, and `length_scale` adjusts the speaking rate. These parameters influence the quality and duration of the generated audio.

```python
## Number of ODE Solver steps
n_timesteps = 10

## Changes to the speaking rate
length_scale=1.0
```

--------------------------------

### Batch Voice Synthesis CLI - BreezyVoice

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Command-line interface for processing multiple text-to-speech requests efficiently from a CSV file. Requires a CSV file path, folder for speaker prompts, output folder for audio, and optionally a model path. Processes requests in bulk.

```bash
python batch_inference.py \
  --csv_file ./data/batch_files.csv \
  --speaker_prompt_audio_folder ./data \
  --output_audio_folder ./results \
  --model_path "MediaTek-Research/BreezyVoice"
```

--------------------------------

### Synthesize Speech with Trained Matcha-TTS Model

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/README.md

This command synthesizes speech from a custom-trained Matcha-TTS model. It requires the text to be synthesized and the path to the model checkpoint.

```bash
matcha-tts --text "<INPUT TEXT>" --checkpoint_path <PATH TO CHECKPOINT>
```

--------------------------------

### Python Base CosyVoice Library Inference Modes

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Demonstrates various inference modes of the base CosyVoice library, including supervised fine-tuning, zero-shot cloning, cross-lingual synthesis, and instruction-based synthesis.

```python
from cosyvoice.cli.cosyvoice import CosyVoice
from cosyvoice.utils.file_utils import load_wav
import torchaudio

# Initialize model
cosyvoice = CosyVoice("MediaTek-Research/BreezyVoice")

# 1. Supervised Fine-Tuning mode (predefined speakers)
available_speakers = cosyvoice.list_avaliable_spks()
print(f"Available speakers: {available_speakers}")

output = cosyvoice.inference_sft(
    tts_text="今天天氣很好",
    spk_id="speaker_001"
)
torchaudio.save("sft_output.wav", output['tts_speech'], 22050)

# 2. Zero-shot voice cloning
prompt_speech = load_wav("reference_speaker.wav", 16000)
output = cosyvoice.inference_zero_shot(
    tts_text="這是使用零樣本語音克隆生成的語音",
    prompt_text="這是參考語音的文本內容",
    prompt_speech_16k=prompt_speech
)
torchaudio.save("zero_shot_output.wav", output['tts_speech'], 22050)

# 3. Cross-lingual synthesis (speak Chinese with English speaker's voice)
english_prompt = load_wav("english_speaker.wav", 16000)
output = cosyvoice.inference_cross_lingual(
    tts_text="用英語說話者的聲音說中文",
    prompt_speech_16k=english_prompt
)
torchaudio.save("cross_lingual_output.wav", output['tts_speech'], 22050)

# 4. Instruction-based synthesis (for instruct models)
output = cosyvoice.inference_instruct(
    tts_text="歡迎來到台灣",
    spk_id="speaker_001",
    instruct_text="用溫柔的語氣說話"  # Instruction for style control
)
torchaudio.save("instruct_output.wav", output['tts_speech'], 22050)
```

--------------------------------

### Single Voice Synthesis CLI - BreezyVoice

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Command-line tool for synthesizing speech from text using custom voice prompts. Supports manual transcription, automatic transcription via Whisper, phonetic correction using bopomofo, and custom model paths. Requires audio prompt, text, and an output path.

```bash
python3 single_inference.py \
  --speaker_prompt_audio_path "data/example.wav" \
  --speaker_prompt_text_transcription "在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。只有擁有解密方法的對象，經由解密過程，才能將密文還原為正常可讀的內容。" \
  --content_to_synthesize "歡迎使用聯發創新基地 BreezyVoice 模型。" \
  --output_path results/out.wav

python3 single_inference.py \
  --speaker_prompt_audio_path "./data/example.wav" \
  --content_to_synthesize "今天天氣真好"

python3 single_inference.py \
  --speaker_prompt_audio_path "./data/example.wav" \
  --content_to_synthesize "今天天氣真好[:ㄏㄠ3]" \
  --speaker_prompt_text_transcription "在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。"

python3 single_inference.py \
  --model_path "MediaTek-Research/BreezyVoice" \
  --speaker_prompt_audio_path "./data/example.wav" \
  --content_to_synthesize "冷氣團南下 北部轉涼白天氣溫降8度" \
  --output_path "./results/weather.wav"
```

--------------------------------

### ONNX Model Export and Inference

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/matcha_tts.egg-info/SOURCES.txt

Provides functionality to export the trained Matcha TTS model to the ONNX format for efficient inference and to perform inference using an ONNX model. This enables deployment on various platforms.

```python
import torch
import onnx

from matcha.models.matcha_tts import MatchaTTS

def export_onnx(model: MatchaTTS, output_path: str, input_example: torch.Tensor):
    """Exports the MatchaTTS model to ONNX format."""
    torch.onnx.export(
        model,
        input_example,
        output_path,
        verbose=True,
        opset_version=14,
        input_names=['input_ids'],
        output_names=['mel_output']
    )
    print(f"Model exported to {output_path}")

def infer_onnx(onnx_path: str, input_data: torch.Tensor):
    """Performs inference using an ONNX model."""
    onnx_model = onnx.load(onnx_path)
    onnx.checker.check_model(onnx_model)

    # Using ONNX Runtime for inference
    import onnxruntime
    session = onnxruntime.InferenceSession(onnx_path)
    input_name = session.get_inputs()[0].name
    output_name = session.get_outputs()[0].name

    predictions = session.run([output_name], {input_name: input_data.cpu().numpy()})
    return torch.tensor(predictions[0])

```

--------------------------------

### Run Batch Inference Script (Bash)

Source: https://github.com/mtkresearch/breezyvoice/blob/main/README.md

Executes the `batch_inference.py` script for processing multiple synthesis tasks from a CSV file. Requires paths to the CSV file, speaker audio folder, and output audio folder. The script synthesizes speech for each entry in the CSV.

```bash
bash run_batch_inference.sh
```

```bash
python batch_inference.py \
  --csv_file ./data/batch_files.csv \
  --speaker_prompt_audio_folder ./data \
  --output_audio_folder ./results
```

--------------------------------

### CLI Batch Processing

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Process multiple TTS requests efficiently using a CSV file for bulk synthesis.

```APIDOC
## CLI Batch Processing

### Description
Process multiple TTS requests efficiently using a CSV file for bulk synthesis.

### Method
CLI

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### CLI Command
```bash
python batch_inference.py \
  --csv_file ./data/batch_files.csv \
  --speaker_prompt_audio_folder ./data \
  --output_audio_folder ./results \
  --model_path "MediaTek-Research/BreezyVoice"
```

### CSV File Format (`data/batch_files.csv`)
```csv
speaker_prompt_audio_filename,speaker_prompt_text_transcription,content_to_synthesize,output_audio_filename
example,在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。只有擁有解密方法的對象，經由解密過程，才能將密文還原為正常可讀的內容。,歡迎使用聯發創新基地 BreezyVoice 模型。,out-BreezyVoice
example,在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。只有擁有解密方法的對象，經由解密過程，才能將密文還原為正常可讀的內容。,今天天氣真好,out-weather
example,在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。,台北101大樓是台灣最高的建築,out-taipei
```
```

--------------------------------

### Synthesizing Speech and Calculating RTF in Python

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

This Python code iterates through a list of texts, synthesizes speech for each, and calculates the Real-Time Factor (RTF) using HiFi-GAN. It prints detailed intermediate results, displays the synthesized waveform, and saves it to a specified folder. Dependencies include `tqdm`, `datetime`, `ipd` (likely IPython.display), and custom functions like `synthesise`, `to_waveform`, and `save_to_folder`. The output includes original text, phonetized text, and RTF values.

```python
temperature = 0.667

outputs, rtfs = [], []
rtfs_w = []
for i, text in enumerate(tqdm(texts)):
    output = synthesise(text) #, torch.tensor([15], device=device, dtype=torch.long).unsqueeze(0))
    output['waveform'] = to_waveform(output['mel'], vocoder)

    # Compute Real Time Factor (RTF) with HiFi-GAN
    t = (dt.datetime.now() - output['start_t']).total_seconds()
    rtf_w = t * 22050 / (output['waveform'].shape[-1])

    ## Pretty print
    print(f"{'*' * 53}")
    print(f"Input text - {i}")
    print(f"{'-' * 53}")
    print(output['x_orig'])
    print(f"{'*' * 53}")
    print(f"Phonetised text - {i}")
    print(f"{'-' * 53}")
    print(output['x_phones'])
    print(f"{'*' * 53}")
    print(f"RTF:\t\t{output['rtf']:.6f}")
    print(f"RTF Waveform:\t{rtf_w:.6f}")
    rtfs.append(output['rtf'])
    rtfs_w.append(rtf_w)

    ## Display the synthesised waveform
    ipd.display(ipd.Audio(output['waveform'], rate=22050))

    ## Save the generated waveform
    save_to_folder(i, output, OUTPUT_FOLDER)

print(f"Number of ODE steps: {n_timesteps}")
print(f"Mean RTF:\t\t\t\t{np.mean(rtfs):.6f} ± {np.std(rtfs):.6f}")
print(f"Mean RTF Waveform (incl. vocoder):\t{np.mean(rtfs_w):.6f} ± {np.std(rtfs_w):.6f}")
```

--------------------------------

### Save Generated Audio using Torchaudio

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Saves synthesized audio to a WAV file using the torchaudio library. It takes the audio tensor, filename, sample rate, and format as input.

```python
import torchaudio

torchaudio.save(
    "output.wav",
    audio_22k,
    22050,
    format="wav"
)
```

--------------------------------

### Text-to-Speech Synthesis using BreezyVoice

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Synthesizes speech from text using the BreezyVoice model. It requires the content to synthesize, an output path, and optionally uses a custom CosyVoice model and a Bopomofo converter. Transcription can be auto-generated if not provided.

```python
from single_inference import single_inference, CustomCosyVoice
from g2pw import G2PWConverter

cosyvoice = CustomCosyVoice("MediaTek-Research/BreezyVoice")
converter = G2PWConverter()

# Transcription is generated automatically if not provided
single_inference(
    speaker_prompt_audio_path="./data/speaker_sample.wav",
    content_to_synthesize="要合成的文字內容",
    output_path="./results/output.wav",
    cosyvoice=cosyvoice,
    bopomofo_converter=converter,
    speaker_prompt_text_transcription=None  # Will auto-transcribe
)
```

--------------------------------

### CLI Single Voice Synthesis

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Generate speech from text using the command-line interface. Supports manual transcription, automatic transcription via Whisper, phonetic correction with bopomofo, and custom model usage.

```APIDOC
## CLI Single Voice Synthesis

### Description
Generate speech from text using the command-line interface. Supports manual transcription, automatic transcription via Whisper, phonetic correction with bopomofo, and custom model usage.

### Method
CLI

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### CLI Commands
```bash
# Basic usage with manual transcription
python3 single_inference.py \
  --speaker_prompt_audio_path "data/example.wav" \
  --speaker_prompt_text_transcription "在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。只有擁有解密方法的對象，經由解密過程，才能將密文還原為正常可讀的內容。" \
  --content_to_synthesize "歡迎使用聯發創新基地 BreezyVoice 模型。" \
  --output_path results/out.wav

# With automatic transcription (Whisper)
python3 single_inference.py \
  --speaker_prompt_audio_path "./data/example.wav" \
  --content_to_synthesize "今天天氣真好"

# With manual phonetic correction using bopomofo
python3 single_inference.py \
  --speaker_prompt_audio_path "./data/example.wav" \
  --content_to_synthesize "今天天氣真好[:ㄏㄠ3]" \
  --speaker_prompt_text_transcription "在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。"

# Using custom model
python3 single_inference.py \
  --model_path "MediaTek-Research/BreezyVoice" \
  --speaker_prompt_audio_path "./data/example.wav" \
  --content_to_synthesize "冷氣團南下 北部轉涼白天氣溫降8度" \
  --output_path "./results/weather.wav"
```
```

--------------------------------

### Configure Autoreload and Matplotlib Backend

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/synthesis.ipynb

Enables automatic reloading of modules during development and sets Matplotlib to render plots inline within the notebook. This facilitates iterative development by reflecting code changes without kernel restarts.

```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
```

--------------------------------

### REST API /v1/audio/speech

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

Generate speech from input text using the specified model and voice.

```APIDOC
### POST /v1/audio/speech

#### Description
Generate speech from input text using the specified model and voice.

#### Method
POST

#### Endpoint
`/v1/audio/speech`

#### Parameters
##### Query Parameters
None

##### Request Body
- **model** (string) - Required - The model to use for speech synthesis (e.g., "tts-1").
- **input** (string) - Required - The text to synthesize.
- **voice** (string) - Optional - The voice to use for synthesis (e.g., "alloy").
- **response_format** (string) - Optional - The format of the output audio (e.g., "wav").
- **speed** (number) - Optional - The speed of the synthesized speech (default: 1.0).

#### Request Example (cURL)
```bash
curl -X POST http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "今天天氣真好",
    "response_format": "wav",
    "speed": 1.0
  }' \
  --output speech.wav
```

#### Request Example (Python OpenAI SDK)
```python
from pathlib import Path
import openai

# Initialize client
client = openai.Client(
    base_url="http://localhost:8080",
    api_key="sk-template"
)

# Generate speech
speech_file_path = Path(__file__).parent / "./results/speech.wav"
response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="冷氣團南下 北部轉涼白天氣溫降8度"
)

# Save audio
with open(speech_file_path, "wb") as audio_file:
    audio_file.write(response.content)
```

#### Response
##### Success Response (200)
- **content** (bytes) - The audio data in the requested format.

#### Response Example
(Audio file content would be here, not representable in JSON text)
```

--------------------------------

### Complete Inference Function for BreezyVoice

Source: https://context7.com/mtkresearch/breezyvoice/llms.txt

A high-level function that orchestrates the entire text-to-speech synthesis process. It initializes the model and converter once for efficiency and includes error handling for robust execution. It supports processing multiple requests sequentially.

```python
from single_inference import single_inference, CustomCosyVoice
from g2pw import G2PWConverter

# Initialize model and converter once (reuse for multiple requests)
cosyvoice = CustomCosyVoice("MediaTek-Research/BreezyVoice")
bopomofo_converter = G2PWConverter()

# Perform complete inference with error handling
try:
    single_inference(
        speaker_prompt_audio_path="./data/example.wav",
        content_to_synthesize="歡迎使用BreezyVoice[:yao1]語音[:yu3 yin1]合成[:he2 cheng2]系統[:xi4 tong3]",
        output_path="./results/output.wav",
        cosyvoice=cosyvoice,
        bopomofo_converter=bopomofo_converter,
        speaker_prompt_text_transcription="在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。"
    )
    print("Synthesis completed successfully!")
except Exception as e:
    print(f"Synthesis failed: {e}")

# Process multiple requests efficiently (reuse model)
requests = [
    ("今天天氣很好", "weather.wav"),
    ("明天會下雨", "tomorrow.wav"),
    ("週末去哪裡玩", "weekend.wav")
]

for content, output_file in requests:
    single_inference(
        speaker_prompt_audio_path="./data/example.wav",
        content_to_synthesize=content,
        output_path=f"./results/{output_file}",
        cosyvoice=cosyvoice,
        bopomofo_converter=bopomofo_converter,
        speaker_prompt_text_transcription="在密碼學中，加密是將明文資訊改變為難以讀取的密文內容，使之不可讀的方法。"
    )
```

--------------------------------

### Update Data Statistics in YAML Configuration

Source: https://github.com/mtkresearch/breezyvoice/blob/main/third_party/Matcha-TTS/README.md

This snippet shows how to update the `mel_mean` and `mel_std` values in the `configs/data/ljspeech.yaml` file. These statistics are computed for the LJSpeech dataset and are crucial for model training.

```yaml
data_statistics:  # Computed for ljspeech dataset
  mel_mean: -5.536622
  mel_std: 2.116101
```