### Install WeText Frontend

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/configuration.md

Install the WeText frontend as a fallback option if ttsfrd is unavailable. This is a simple pip installation.

```bash
pip install wetext
```

--------------------------------

### Run Basic Example Script

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Execute the main example script to see basic CosyVoice functionality. Ensure you have the necessary environment set up.

```sh
python example.py
```

--------------------------------

### Install ttsfrd Frontend Dependencies

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/configuration.md

Install the necessary dependencies for the ttsfrd text frontend. This includes unzipping resource files and installing Python wheels.

```bash
cd pretrained_models/CosyVoice-ttsfrd
unzip resource.zip -d .
pip install ttsfrd_dependency-0.1-py3-none-any.whl
pip install ttsfrd-0.4.2-cp310-cp310-linux_x86_64.whl
```

--------------------------------

### Set Up Python Environment

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/README.md

Create and activate a Conda environment for CosyVoice with Python 3.10 and install project dependencies.

```bash
conda create -n cosyvoice python=3.10
conda activate cosyvoice
pip install -r requirements.txt
```

--------------------------------

### Start Triton and TensorRT-LLM Deployment

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Quickly start the Triton Inference Server with TensorRT-LLM backend for CosyVoice deployment using Docker Compose. This setup can provide significant acceleration.

```sh
cd runtime/triton_trtllm
docker compose up -d
```

--------------------------------

### Run vLLM Example Script

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Execute the example script specifically designed for vLLM inference. This script utilizes the vLLM engine for CosyVoice models.

```sh
python vllm_example.py
```

--------------------------------

### Example Usage of inference_instruct2

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Demonstrates how to use the `inference_instruct2` method for synthesizing speech with specific dialect instructions. The output speech is saved to a WAV file.

```python
for output in cosyvoice.inference_instruct2(
    '收到好友的礼物',
    '用四川话说这句话<|endofprompt|>',
    'reference.wav'
):
    torchaudio.save('sichuan.wav', output['tts_speech'], cosyvoice.sample_rate)
```

--------------------------------

### Install Text Normalization Dependencies

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/errors.md

Install necessary dependencies for text normalization. This includes commands for installing 'ttsfrd' for best performance or 'wetext' as a fallback option.

```bash
# For best performance
cd pretrained_models/CosyVoice-ttsfrd
unzip resource.zip -d .
pip install ttsfrd_dependency-0.1-py3-none-any.whl
pip install ttsfrd-0.4.2-cp310-cp310-linux_x86_64.whl

# Or fallback
pip install wetext
```

--------------------------------

### CosyVoice gRPC Client Example (Python)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Example Python code demonstrating how to connect to the gRPC server and perform an inference request.

```APIDOC
## Example (Python gRPC Client)

### Description
Provides a Python code snippet for connecting to the CosyVoice gRPC server and making an inference call.

### Client Code

```python
import grpc
import numpy as np
import cosyvoice_pb2
import cosyvoice_pb2_grpc

# Connect
channel = grpc.insecure_channel('localhost:50000')
stub = cosyvoice_pb2_grpc.CosyVoiceServicer(channel)

# SFT inference
request = cosyvoice_pb2.Request()
request.sft_request.tts_text = '你好'
request.sft_request.spk_id = '中文女'

audio_chunks = []
for response in stub.Inference(request):
    audio_chunks.append(response.tts_audio)

# Combine audio chunks and save
import wave
audio = np.frombuffer(b''.join(audio_chunks), dtype=np.int16)
# ... save to WAV file
```

### Audio Format

- Encoding: PCM int16
- Sample rate: Model-dependent (typically 22050 Hz for CosyVoice1, 24000 Hz for CosyVoice2/3)
- Channels: Mono
- Byte order: Native endianness
```

--------------------------------

### Launch Disaggregated Servers (Stage 7)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.DiT.md

Run stage 7 to launch servers in a disaggregated setup, placing the LLM on GPU 0 and Token2Wav servers on GPUs 1-3.

```sh
bash run_stepaudio2_dit_token2wav.sh 7 7
```

--------------------------------

### Run FastAPI Server

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Starts the FastAPI server for CosyVoice2 inference. Specify the port and model directory.

```bash
python runtime/python/fastapi/server.py \
    --port 50000 \
    --model_dir pretrained_models/CosyVoice2-0.5B
```

--------------------------------

### Install Compatible vLLM Versions

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/errors.md

If you encounter an 'ImportError: vLLM Not Installed' when `load_vllm=True`, install a compatible vLLM version using pip.

```bash
# Install compatible vLLM versions
pip install vllm==0.9.0     # Tested version
# OR
pip install vllm==0.11.0    # Newer version
```

--------------------------------

### Run gRPC Server

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Starts the gRPC server for CosyVoice2. Configure the port, maximum concurrency, and model directory.

```bash
python runtime/python/grpc/server.py \
    --port 50000 \
    --max_conc 4 \
    --model_dir pretrained_models/CosyVoice2-0.5B
```

--------------------------------

### Install ttsfrd Package

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Unzip the ttsfrd resource and install the ttsfrd package and its dependencies for improved text normalization performance. This step is optional.

```sh
cd pretrained_models/CosyVoice-ttsfrd/
unzip resource.zip -d .
pip install ttsfrd_dependency-0.1-py3-none-any.whl
pip install ttsfrd-0.4.2-cp310-cp310-linux_x86_64.whl
```

--------------------------------

### Run Specific Stages of the `run.sh` Script

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.Unet.md

Execute a subset of stages from the `run.sh` script by specifying the start and stop stages. This allows for granular control over the model export and server setup process.

```sh
bash run.sh <start_stage> <stop_stage> [service_type]
```

--------------------------------

### Install TensorRT or Disable

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/errors.md

Instructions for installing TensorRT via pip or disabling its use by setting `load_trt=False` during CosyVoice initialization.

```bash
# Install TensorRT (requires NVIDIA account)
pip install tensorrt
```

```python
# Or disable TensorRT
cosyvoice = CosyVoice(model_dir, load_trt=False)
```

--------------------------------

### CosyVoice Instruction Format Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Demonstrates the instruction format for CosyVoice 2/3, ending with the required '<|endofprompt|>' token.

```text
"用四川话说这句话<|endofprompt|>"
"请用广东话表达。<|endofprompt|>"
```

--------------------------------

### Python gRPC Client Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Demonstrates how to connect to the gRPC server and perform a SFT inference request using Python. Handles combining audio chunks received from the server.

```python
import grpc
import numpy as np
import cosyvoice_pb2
import cosyvoice_pb2_grpc

# Connect
channel = grpc.insecure_channel('localhost:50000')
stub = cosyvoice_pb2_grpc.CosyVoiceServicer(channel)

# SFT inference
request = cosyvoice_pb2.Request()
request.sft_request.tts_text = '你好'
request.sft_request.spk_id = '中文女'

audio_chunks = []
for response in stub.Inference(request):
    audio_chunks.append(response.tts_audio)

# Combine audio chunks and save
import wave
audio = np.frombuffer(b''.join(audio_chunks), dtype=np.int16)
# ... save to WAV file
```

--------------------------------

### Install CosyVoice Dependencies and Models

Source: https://github.com/funaudiollm/cosyvoice/blob/main/FAQ.md

To resolve 'cannot find resource.zip' or 'cannot unzip resource.zip' errors, ensure git-lfs is installed. Then, clone the model repository, unzip the resource file, and install the necessary wheel package.

```sh
git clone https://www.modelscope.cn/iic/CosyVoice-ttsfrd.git pretrained_models/CosyVoice-ttsfrd
cd pretrained_models/CosyVoice-ttsfrd/
unzip resource.zip -d .
pip install ttsfrd-0.3.6-cp38-cp38-linux_x86_64.whl
```

--------------------------------

### Zero-Shot Voice Cloning - Initial Setup

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Perform zero-shot voice cloning by providing a reference audio file and text. This snippet also shows how to save the cloned voice for future use.

```python
# First time: provide reference audio info
for output in cosyvoice.inference_zero_shot(
    '你好，我是AI',
    '这是参考文本',
    'reference_audio.wav'
):
    torchaudio.save('output.wav', output['tts_speech'], cosyvoice.sample_rate)

# Save for reuse
cosyvoice.add_zero_shot_spk(
    '这是参考文本',
    'reference_audio.wav',
    'my_voice'
)
cosyvoice.save_spkinfo()
```

--------------------------------

### Start CosyVoice Web UI

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Launch the web user interface for CosyVoice. Specify the port and the directory containing the model weights. This allows for quick familiarization with CosyVoice.

```python
# change iic/CosyVoice-300M-SFT for sft inference, or iic/CosyVoice-300M-Instruct for instruct inference
python3 webui.py --port 50000 --model_dir pretrained_models/CosyVoice-300M
```

--------------------------------

### Model Configuration (YAML)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/types.md

Example structure for model configuration files (cosyvoice.yaml, cosyvoice2.yaml, cosyvoice3.yaml). Defines parameters for LLM, Flow, and Vocoder modules.

```yaml
llm:
  # LLM module configuration
  text_encoder_input_size: 512
  llm_input_size: 768
  text_token_size: 4096
  speech_token_size: 4096

flow:
  # Flow model configuration
  input_size: 512
  output_size: 80
  input_frame_rate: 50
  vocab_size: 4096

hift:
  # Vocoder configuration
  hidden_size: 256

sample_rate: 22050
allowed_special: "all"
```

--------------------------------

### Launch OpenAI-Compatible Server

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.Unet.md

Commands to clone the repository, install dependencies, and run the FastAPI bridge for an OpenAI-compatible TTS server. Note that only offline TTS mode is currently supported.

```sh
git clone https://github.com/yuekaizhang/Triton-OpenAI-Speech.git
cd Triton-OpenAI-Speech
pip install -r requirements.txt

# After the Triton service is running, start the FastAPI bridge:
python3 tts_server.py --url http://localhost:8000 --ref_audios_dir ./ref_audios/ --port 10086 --default_sample_rate 24000
```

```sh
bash test/test_cosyvoice.sh
```

--------------------------------

### Inference Instruct (CosyVoice 1.0) - Curl Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Performs instruction-based synthesis for CosyVoice 1.0 using curl. Requires tts_text, spk_id, and instruct_text.

```bash
curl -X POST http://localhost:50000/inference_instruct \
  -F "tts_text=挑战来临" \
  -F "spk_id=中文男" \
  -F "instruct_text=Speak with courage and determination" \
  --output output.wav
```

--------------------------------

### Download CosyVoice Models

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Download pre-trained models for CosyVoice using the modelscope library. Ensure you have the library installed.

```python
from modelscope import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
snapshot_download('iic/CosyVoice2-0.5B', local_dir='pretrained_models/CosyVoice2-0.5B')
snapshot_download('iic/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Create a Conda environment named 'cosyvoice' with Python 3.10, activate it, and install project dependencies from 'requirements.txt'. Includes commands for Ubuntu and CentOS to install 'sox' if compatibility issues arise.

```sh
conda create -n cosyvoice -y python=3.10
conda activate cosyvoice
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

# If you encounter sox compatibility issues
# ubuntu
sudo apt-get install sox libsox-dev
# centos
sudo yum install sox sox-devel
```

--------------------------------

### Install vLLM and Dependencies for CosyVoice

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Install vLLM (version 0.9.0 or 0.11.x+) and compatible transformers and numpy versions. This is recommended for efficient CosyVoice inference. A new conda environment is suggested to avoid conflicts.

```sh
conda create -n cosyvoice_vllm --clone cosyvoice
conda activate cosyvoice_vllm
# for vllm==0.9.0
pip install vllm==v0.9.0 transformers==4.51.3 numpy==1.26.4 -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
# for vllm>=0.11.0
pip install vllm==v0.11.0 transformers==4.57.1 numpy==1.26.4 -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
```

--------------------------------

### Export Models and Launch Triton Server

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice3.md

Run stages 0 through 3 within the Docker container to prepare models and start the Triton server.

```sh
# This command runs stages 0, 1, 2, and 3
bash run_cosyvoice3.sh 0 3
```

--------------------------------

### Inference Instruct2 (CosyVoice 2/3) - Curl Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Performs instruction-based synthesis with zero-shot voice cloning for CosyVoice 2/3 using curl. Requires tts_text, instruct_text, and a prompt_wav file.

```bash
curl -X POST http://localhost:50000/inference_instruct2 \
  -F "tts_text=收到好友的礼物" \
  -F "instruct_text=用四川话说这句话<|endofprompt|>" \
  -F "prompt_wav=@reference.wav" \
  --output output.wav
```

--------------------------------

### inference_instruct()

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Performs instruction-based voice synthesis. It takes text, a speaker ID, and an instruction to guide the synthesis. GPU is optional, and streaming is supported.

```APIDOC
## inference_instruct()

### Description
Performs instruction-based voice synthesis. It takes text, a speaker ID, and an instruction to guide the synthesis. GPU is optional, and streaming is supported.

### Input
- text (string)
- speaker_id (string)
- instruction (string)

### Output
- audio (audio file)

### GPU
Optional

### Stream
Supported
```

--------------------------------

### Use Tested vLLM Versions

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/errors.md

To resolve 'Version Incompatibility: vLLM' errors, uninstall the current vLLM and install a tested version like 0.9.0 or 0.11.0+.

```bash
# Use tested versions only
pip uninstall vllm
pip install vllm==0.9.0  # or 0.11.0+, not 0.10.x
```

--------------------------------

### Inference Cross-Lingual Synthesis - Curl Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Performs cross-lingual zero-shot synthesis using curl. Clones a voice from a reference audio and synthesizes text in a different language.

```bash
curl -X POST http://localhost:50000/inference_cross_lingual \
  -F "tts_text=<|en|>Hello world" \
  -F "prompt_wav=@chinese_speaker.wav" \
  --output english_output.wav
```

--------------------------------

### Run Specific Stages of the Workflow Script

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.DiT.md

Execute a subset of the `run_stepaudio2_dit_token2wav.sh` script by specifying start and stop stages. This is useful for debugging or running specific parts of the pipeline.

```sh
bash run_stepaudio2_dit_token2wav.sh <start_stage> <stop_stage>
```

--------------------------------

### Benchmark with Offline Inference Mode

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.Unet.md

Execute stage 6 of the `run.sh` script for benchmarking offline inference. This stage may require installing FlashCosyVoice for token2wav batching and downloading specific ONNX model files.

```sh
# install FlashCosyVoice for token2wav batching
# git clone https://github.com/yuekaizhang/FlashCosyVoice.git /workspace/FlashCosyVoice -b trt
# cd /workspace/FlashCosyVoice
# pip install -e .
# cd -
# wget https://huggingface.co/cosyvoice2_flow_onnx/resolve/main/flow.decoder.estimator.fp32.dynamic_batch.onnx -O $model_scope_model_local_dir/flow.decoder.estimator.fp32.dynamic_batch.onnx

bash run.sh 6 6
```

```sh
# You can also switch to huggingface backend by setting backend=hf
```

--------------------------------

### frontend_instruct2

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Prepares input tensors for instruction and zero-shot synthesis (CosyVoice 2/3 only). Combines instruction-based synthesis with zero-shot voice cloning capabilities.

```APIDOC
## frontend_instruct2(tts_text, instruct_text, prompt_wav, resample_rate, zero_shot_spk_id)

### Description
Prepares input tensors for instruction-based synthesis combined with zero-shot voice cloning, specifically for CosyVoice 2/3. It allows for guided synthesis using instructions and a reference voice.

### Parameters
- **tts_text** (str) - The text to be synthesized.
- **instruct_text** (str) - The instruction text to guide the synthesis.
- **prompt_wav** (str) - Path to the reference audio (16 kHz WAV).
- **resample_rate** (int) - The target sample rate (e.g., 24000 for CosyVoice2/3).
- **zero_shot_spk_id** (str) - The ID for the zero-shot speaker. Can use a cached profile if available.

### Returns
Dict[str, torch.Tensor] - A dictionary containing the necessary tensors for instruction and zero-shot synthesis.
```

--------------------------------

### Benchmark Disaggregated Server Configuration (Stage 8)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.DiT.md

Execute stage 8 to run the benchmark client for the disaggregated server configuration.

```sh
bash run_stepaudio2_dit_token2wav.sh 8 8
```

--------------------------------

### FastAPI Error Response Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

FastAPI errors are returned in JSON format. This example shows a typical error for a missing parameter.

```json
{
  "detail": "Missing required parameter: tts_text"
}
```

--------------------------------

### frontend_instruct

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Prepares input tensors for instruction-based synthesis (CosyVoice 1.0 only). It takes text, speaker ID, and instruction text as input.

```APIDOC
## frontend_instruct(tts_text, spk_id, instruct_text)

### Description
Prepares input tensors for instruction-based synthesis, specifically for CosyVoice 1.0. It takes the target text, speaker ID, and an instruction text to guide the synthesis.

### Parameters
- **tts_text** (str) - The text to be synthesized.
- **spk_id** (str) - The speaker ID.
- **instruct_text** (str) - The instruction text to guide the synthesis.

### Returns
Dict[str, torch.Tensor] - A dictionary containing the necessary tensors for instruction-based synthesis.
```

--------------------------------

### Export Models and Launch Triton Server (Stages 0-3)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.DiT.md

This command executes stages 0 through 3 of the `run_stepaudio2_dit_token2wav.sh` script, which prepares models and launches the Triton server.

```sh
# This command runs stages 0, 1, 2, and 3
bash run_stepaudio2_dit_token2wav.sh 0 3
```

--------------------------------

### Launch gRPC Server

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Launches the gRPC server with specified port, maximum concurrency, and model directory. Ensure the model directory contains the necessary model files.

```bash
python runtime/python/grpc/server.py --port 50000 --max_conc 4 --model_dir iic/CosyVoice2-0.5B
```

--------------------------------

### Build and Run CosyVoice Docker Images

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Build the CosyVoice Docker image and run instances for the FastAPI and gRPC servers. Ensure CUDA is available for GPU acceleration.

```bash
cd runtime/python
docker build -t cosyvoice:v1.0 .

# FastAPI server
docker run -d --runtime=nvidia -p 50000:50000 cosyvoice:v1.0 \
  /bin/bash -c "cd /opt/CosyVoice/runtime/python/fastapi && \
  python3 server.py --port 50000 --model_dir iic/CosyVoice2-0.5B"

# gRPC server
docker run -d --runtime=nvidia -p 50000:50000 cosyvoice:v1.0 \
  /bin/bash -c "cd /opt/CosyVoice/runtime/python/grpc && \
  python3 server.py --port 50000 --model_dir iic/CosyVoice2-0.5B"
```

--------------------------------

### Launch FastAPI Server with TensorRT Acceleration

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Use this command to launch the FastAPI server with TensorRT acceleration enabled. Requires an NVIDIA GPU and CUDA environment.

```bash
# Launch with TensorRT acceleration
python runtime/python/fastapi/server.py \
  --model_dir pretrained_models/CosyVoice2-0.5B \
  --load_trt \
  --fp16
```

--------------------------------

### Run Subset of run_cosyvoice3.sh Stages

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice3.md

Execute specific stages of the CosyVoice3 workflow by providing start and stop stage numbers.

```sh
bash run_cosyvoice3.sh <start_stage> <stop_stage>
```

--------------------------------

### Prepare Inputs for Instruction + Zero-Shot Synthesis (CosyVoice 2/3)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Prepares input for instruction-based synthesis combined with zero-shot voice cloning, specifically for CosyVoice versions 2 and 3.

```python
def frontend_instruct2(self,
                       tts_text: str,
                       instruct_text: str,
                       prompt_wav: str,
                       resample_rate: int,
                       zero_shot_spk_id: str) -> Dict[str, torch.Tensor]
```

--------------------------------

### Inference SFT (Speaker-based TTS) - Curl Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Performs speaker-based text-to-speech synthesis using curl. Requires tts_text and spk_id.

```bash
curl -X POST http://localhost:50000/inference_sft \
  -F "tts_text=你好，我是AI助手" \
  -F "spk_id=中文女" \
  --output output.wav
```

--------------------------------

### CosyVoice3 Constructor Parameters

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/configuration.md

Initialize CosyVoice3, which removes the JIT loading option. vLLM is preferred for LLM acceleration. Note that fp16 mode on the DiT engine has performance issues.

```python
CosyVoice3(
    model_dir: str,
    load_trt: bool = False,
    load_vllm: bool = False,
    fp16: bool = False,
    trt_concurrent: int = 1
)
```

--------------------------------

### Inference SFT (Speaker-based TTS) - Python Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Performs speaker-based text-to-speech synthesis using Python requests. Saves the streamed audio to output.wav.

```python
import requests

response = requests.post(
    'http://localhost:50000/inference_sft',
    data={'tts_text': '你好', 'spk_id': '中文女'},
    stream=True
)
with open('output.wav', 'wb') as f:
    f.write(response.content)
```

--------------------------------

### Export Models and Launch Triton Server (Stages 0-3)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.Unet.md

This command executes stages 0 through 3 of the `run.sh` script, which includes downloading the model, converting it to TensorRT-LLM format, creating the Triton model repository, and launching the Triton Inference Server.

```sh
# This command runs stages 0, 1, 2, and 3
bash run.sh 0 3
```

--------------------------------

### Prepare Inputs for Zero-Shot Voice Cloning

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Prepares inputs for zero-shot voice cloning. Extracts embeddings and speech tokens from a reference audio prompt.

```python
def frontend_zero_shot(self,
                       tts_text: str,
                       prompt_text: str,
                       prompt_wav: str,
                       resample_rate: int,
                       zero_shot_spk_id: str) -> Dict[str, torch.Tensor]
```

--------------------------------

### Speed Control - Speed Up Synthesis

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Control the speech synthesis speed by setting the 'speed' parameter. This example demonstrates speeding up the speech to 1.5x the normal speed.

```python
# Speed up (1.5x)
for output in cosyvoice.inference_sft('你好', '中文女', speed=1.5):
    pass
```

--------------------------------

### Prepare Inputs for Instruction-Based Synthesis (CosyVoice 1.0)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Prepares input for instruction-based synthesis specifically for CosyVoice version 1.0.

```python
def frontend_instruct(self,
                      tts_text: str,
                      spk_id: str,
                      instruct_text: str) -> Dict[str, torch.Tensor]
```

--------------------------------

### Speed Control - Slow Down Synthesis

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Control the speech synthesis speed by setting the 'speed' parameter. This example demonstrates slowing down the speech to 0.5x the normal speed.

```python
# Slow down (0.5x speed)
for output in cosyvoice.inference_sft('你好', '中文女', speed=0.5):
    pass
```

--------------------------------

### POST /inference_instruct2

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Provides instruction and zero-shot synthesis for CosyVoice 2/3. It synthesizes speech using text, an instruction, and a reference audio file.

```APIDOC
## POST /inference_instruct2

### Description
Instruction + zero-shot synthesis (CosyVoice2/3 only). This endpoint synthesizes speech by combining an instruction describing the voice/style with a reference audio for voice cloning.

### Method
POST

### Endpoint
/inference_instruct2

### Parameters
#### Request Body
- **tts_text** (string) - Required - Text to synthesize
- **instruct_text** (string) - Required - Instruction describing voice/style
- **prompt_wav** (file) - Required - Reference audio (16 kHz WAV)

### Request Example
```bash
curl -X POST http://localhost:50000/inference_instruct2 \
  -F "tts_text=收到好友的礼物" \
  -F "instruct_text=用四川话说这句话<|endofprompt|>" \
  -F "prompt_wav=@reference.wav" \
  --output output.wav
```

### Response
#### Success Response (200)
- Content-Type: `audio/wav`
- Body: Streaming audio combining instruction and voice cloning

#### Response Example
(Streaming audio/wav)
```

--------------------------------

### Inference Zero Shot (Voice Cloning) - Curl Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Performs zero-shot voice cloning synthesis using curl. Requires tts_text, prompt_text, and a prompt_wav file.

```bash
curl -X POST http://localhost:50000/inference_zero_shot \
  -F "tts_text=这是新的文本" \
  -F "prompt_text=这是参考文本" \
  -F "prompt_wav=@reference.wav" \
  --output output.wav
```

--------------------------------

### Instruction + Voice Cloning (CosyVoice2/3)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Combine instruction-based control with zero-shot voice cloning using CosyVoice2 or CosyVoice3 models. Use the '<|endofprompt|>' tag to separate instructions from other parameters.

```python
cosyvoice = AutoModel(model_dir='pretrained_models/CosyVoice2-0.5B')
for output in cosyvoice.inference_instruct2(
    '你好',
    '用广东话说<|endofprompt|>',  # Instruction with endofprompt tag
    'reference.wav'
):
    torchaudio.save('output.wav', output['tts_speech'], cosyvoice.sample_rate)
```

--------------------------------

### Launch Service with Docker Compose

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice3.md

Use this command to quickly launch the CosyVoice3 service using Docker Compose.

```sh
docker compose -f docker-compose.cosyvoice3.yml up
```

--------------------------------

### Set PYTHONPATH for Matcha-TTS

Source: https://github.com/funaudiollm/cosyvoice/blob/main/FAQ.md

If you encounter a ModuleNotFoundError for 'matcha', ensure Matcha-TTS is correctly installed and its path is added to PYTHONPATH. This allows Python scripts to import modules from the Matcha-TTS directory.

```sh
export PYTHONPATH=third_party/Matcha-TTS
```

--------------------------------

### CosyVoice2 Constructor Parameters

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/configuration.md

Initialize CosyVoice2 with options for JIT, TensorRT, vLLM, and fp16. vLLM requires version 0.9.0 or 0.11.0+ and enables GPU-optimized LLM inference.

```python
CosyVoice2(
    model_dir: str,
    load_jit: bool = False,
    load_trt: bool = False,
    load_vllm: bool = False,
    fp16: bool = False,
    trt_concurrent: int = 1
)
```

--------------------------------

### Check and Use Correct Speaker ID

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/errors.md

When encountering a 'KeyError: Speaker Not Found', use this example to list available speaker IDs and then reference a correct one in your inference call.

```python
# Check available speakers
speakers = cosyvoice.list_available_spks()
print(speakers)  # List all valid IDs

# Use correct speaker ID
if '中文女' in speakers:
    cosyvoice.inference_sft('text', '中文女')
else:
    print(f"Available speakers: {speakers}")
```

--------------------------------

### frontend_vc

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Prepares input tensors for voice conversion. It takes the source speech and a prompt audio file to perform voice conversion.

```APIDOC
## frontend_vc(source_speech_16k, prompt_wav, resample_rate)

### Description
Prepares input tensors for voice conversion. This method takes the source speech (at 16kHz) and a prompt audio file to facilitate voice conversion.

### Parameters
- **source_speech_16k** (str) - Path to the source speech audio (16 kHz).
- **prompt_wav** (str) - Path to the prompt audio (16 kHz WAV).
- **resample_rate** (int) - The target sample rate (e.g., 24000 for CosyVoice2/3).

### Returns
Dict[str, torch.Tensor] - A dictionary containing the necessary tensors for voice conversion.
```

--------------------------------

### Launch CosyVoice FastAPI Server

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Launches the FastAPI server for CosyVoice. Specify the port and the directory containing the models.

```bash
python runtime/python/fastapi/server.py --port 50000 --model_dir pretrained_models/CosyVoice2-0.5B
```

--------------------------------

### CosyVoiceFrontEnd.frontend_instruct2

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/types.md

Combines instruction with zero-shot voice cloning. This function takes various tensor inputs related to text, speech tokens, and embeddings to generate output for text-to-speech with voice cloning.

```APIDOC
## CosyVoiceFrontEnd.frontend_instruct2

### Description
Combines instruction with zero-shot voice cloning. This function takes various tensor inputs related to text, speech tokens, and embeddings to generate output for text-to-speech with voice cloning.

### Function Signature
`CosyVoiceFrontEnd.frontend_instruct2()`

### Input Parameters
- **text** (torch.Tensor) - Input text tensor.
- **text_len** (torch.Tensor) - Length of the input text tensor.
- **flow_prompt_speech_token** (torch.Tensor) - Speech tokens from the flow prompt.
- **flow_prompt_speech_token_len** (torch.Tensor) - Length of the flow prompt speech tokens.
- **prompt_speech_feat** (torch.Tensor) - Speech features from the prompt.
- **prompt_speech_feat_len** (torch.Tensor) - Length of the prompt speech features.
- **llm_embedding** (torch.Tensor) - Language model embedding.
- **flow_embedding** (torch.Tensor) - Flow embedding.
- **prompt_text** (torch.Tensor) - Prompt text tensor.
- **prompt_text_len** (torch.Tensor) - Length of the prompt text tensor.

### Output
(Implicitly returns a structure containing the results of the operation, details not specified in source.)
```

--------------------------------

### CosyVoiceModel.load_jit

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Loads JIT-compiled modules for optimized inference.

```APIDOC
## CosyVoiceModel.load_jit

### Description
Loads JIT-compiled modules for optimized inference.

### Parameters
- **llm_text_encoder_model** (str) - Required
- **llm_llm_model** (str) - Required
- **flow_encoder_model** (str) - Required
```

--------------------------------

### Get Tiktoken Encoding with CosyVoice Special Tokens

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/utils-and-modules.md

Retrieves a tiktoken encoding object, pre-configured with CosyVoice-specific special tokens. Use this function to ensure proper handling of custom tokens during encoding. 

```python
from cosyvoice.tokenizer.tokenizer import get_encoding
enc = get_encoding()
tokens = enc.encode("Hello <|en|>")
```

--------------------------------

### Inference Zero Shot (Voice Cloning) - Python Example

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Performs zero-shot voice cloning synthesis using Python requests. Reads reference audio from reference.wav and saves synthesized audio to output.wav.

```python
import requests

with open('reference.wav', 'rb') as f:
    files = {'prompt_wav': f}
    data = {
        'tts_text': '新的文本',
        'prompt_text': '参考文本'
    }
    response = requests.post(
        'http://localhost:50000/inference_zero_shot',
        files=files,
        data=data,
        stream=True
    )
    with open('output.wav', 'wb') as out:
        out.write(response.content)
```

--------------------------------

### Download Models using Hugging Face SDK

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Download pretrained CosyVoice models and resources using the Hugging Face SDK. Specify the model name and the local directory for saving. This is an alternative for overseas users.

```python
# for oversea users, huggingface SDK model download
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
snapshot_download('FunAudioLLM/CosyVoice2-0.5B', local_dir='pretrained_models/CosyVoice2-0.5B')
snapshot_download('FunAudioLLM/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
snapshot_download('FunAudioLLM/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT')
snapshot_download('FunAudioLLM/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct')
snapshot_download('FunAudioLLM/CosyVoice-ttsfrd', local_dir='pretrained_models/CosyVoice-ttsfrd')
```

--------------------------------

### Handle CUDA Availability and Memory

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/errors.md

Check for CUDA availability and configure CosyVoice accordingly. Monitor GPU memory usage to prevent out-of-memory errors. This example demonstrates conditional initialization based on CUDA support.

```python
# Check CUDA availability
import torch
if not torch.cuda.is_available():
    print("CUDA not available, disabling optimization flags")
    # These will be auto-disabled anyway
    cosyvoice = CosyVoice(model_dir, load_jit=False, load_trt=False, fp16=False)
else:
    cosyvoice = CosyVoice(model_dir, load_jit=True, load_trt=True, fp16=True)

# Monitor memory
print(f"GPU Memory: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
```

--------------------------------

### Build Triton CosyVoice3 Docker Image

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice3.md

Build the Docker image from scratch using the provided Dockerfile.

```sh
docker build . -f Dockerfile.server -t soar97/triton-cosyvoice:25.06
```

--------------------------------

### Instruct Frontend Input Dictionary Structure (CosyVoice 1.0)

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/types.md

This dictionary is the output of the `frontend_instruct` function for CosyVoice 1.0. It includes synthesis text, instruction text, and flow speaker embeddings, with LLM embeddings removed to prevent information leakage.

```python
{
    'text': torch.Tensor,
    'text_len': torch.Tensor,
    'llm_embedding': None,  # Removed to prevent info leakage
    'flow_embedding': torch.Tensor,
    'prompt_text': torch.Tensor,
    'prompt_text_len': torch.Tensor
}
```

--------------------------------

### inference_instruct2()

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/quick-reference.md

Performs a second type of instruction-based voice synthesis. It takes text, an instruction, and an audio sample to generate audio. GPU is optional, and streaming is supported.

```APIDOC
## inference_instruct2()

### Description
Performs a second type of instruction-based voice synthesis. It takes text, an instruction, and an audio sample to generate audio. GPU is optional, and streaming is supported.

### Input
- text (string)
- instruction (string)
- audio (audio file)

### Output
- audio (audio file)

### GPU
Optional

### Stream
Supported
```

--------------------------------

### Launch CosyVoice Solutions with Docker Compose

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.md

Use these Docker Compose commands to launch the different CosyVoice acceleration solutions. Ensure you are in the correct directory for each respective docker-compose file.

```shell
docker compose -f docker-compose.cosyvoice3.yml up
```

```shell
docker compose -f docker-compose.cosyvoice2.unet.yml up
```

```shell
docker compose -f docker-compose.cosyvoice2.dit.yml up
```

--------------------------------

### Launch CosyVoice2 Service with Docker Compose

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice2.Unet.md

Use this command to quickly launch the CosyVoice2 service using a pre-configured Docker Compose file.

```sh
docker compose -f docker-compose.cosyvoice2.unet.yml up
```

--------------------------------

### Prepare Inputs for Voice Conversion

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Prepares input tensors for voice conversion tasks. Requires the source speech audio and a reference prompt audio.

```python
def frontend_vc(self,
                source_speech_16k: str,
                prompt_wav: str,
                resample_rate: int) -> Dict[str, torch.Tensor]
```

--------------------------------

### CosyVoice gRPC Server Launch

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Command to launch the gRPC server with configurable parameters for port, concurrency, and model directory.

```APIDOC
## Server Launch

### Description
Launches the gRPC server for CosyVoice.

### Command
```bash
python runtime/python/grpc/server.py --port 50000 --max_conc 4 --model_dir iic/CosyVoice2-0.5B
```

### Parameters

#### Command Line Arguments

- `--port` (int): Server port. Default: 50000.
- `--max_conc` (int): Maximum concurrent requests. Default: 4.
- `--model_dir` (string): Model path or ModelScope repo ID.
```

--------------------------------

### Instruct2 Frontend Output Structure

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/types.md

This structure represents the output of the `CosyVoiceFrontEnd.frontend_instruct2()` function, combining instruction with zero-shot voice cloning. Fields are a union of instruct and zero-shot types.

```python
{
    'text': torch.Tensor,
    'text_len': torch.Tensor,
    'flow_prompt_speech_token': torch.Tensor,
    'flow_prompt_speech_token_len': torch.Tensor,
    'prompt_speech_feat': torch.Tensor,
    'prompt_speech_feat_len': torch.Tensor,
    'llm_embedding': torch.Tensor,
    'flow_embedding': torch.Tensor,
    'prompt_text': torch.Tensor,
    'prompt_text_len': torch.Tensor
}
```

--------------------------------

### Download Models using ModelScope SDK

Source: https://github.com/funaudiollm/cosyvoice/blob/main/README.md

Download pretrained CosyVoice models and resources using the ModelScope SDK. Specify the model name and the local directory for saving.

```python
# modelscope SDK model download
from modelscope import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
snapshot_download('iic/CosyVoice2-0.5B', local_dir='pretrained_models/CosyVoice2-0.5B')
snapshot_download('iic/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
snapshot_download('iic/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT')
snapshot_download('iic/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct')
snapshot_download('iic/CosyVoice-ttsfrd', local_dir='pretrained_models/CosyVoice-ttsfrd')
```

--------------------------------

### Benchmark Triton Server in Client-Server Mode

Source: https://github.com/funaudiollm/cosyvoice/blob/main/runtime/triton_trtllm/README.Cosyvoice3.md

Execute stage 4 to benchmark the running Triton server. Parameters within the script can be customized.

```sh
bash run_cosyvoice3.sh 4 4

# You can customize parameters such as the number of tasks inside the script.
```

--------------------------------

### Prepare Inputs for Cross-Lingual Synthesis

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/api-reference-main.md

Prepares inputs for cross-lingual text-to-speech synthesis. It removes LLM prompt tokens to facilitate language switching.

```python
def frontend_cross_lingual(self,
                           tts_text: str,
                           prompt_wav: str,
                           resample_rate: int,
                           zero_shot_spk_id: str) -> Dict[str, torch.Tensor]
```

--------------------------------

### POST /inference_instruct

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/endpoints.md

Enables instruction-based synthesis for CosyVoice 1.0. It synthesizes speech based on text, speaker ID, and a natural language instruction.

```APIDOC
## POST /inference_instruct

### Description
Instruction-based synthesis with speaker control (CosyVoice 1.0 only). This endpoint synthesizes speech based on text, a speaker ID, and a natural language instruction.

### Method
POST

### Endpoint
/inference_instruct

### Parameters
#### Request Body
- **tts_text** (string) - Required - Text to synthesize
- **spk_id** (string) - Required - Speaker ID
- **instruct_text** (string) - Required - Natural language instruction

### Request Example
```bash
curl -X POST http://localhost:50000/inference_instruct \
  -F "tts_text=挑战来临" \
  -F "spk_id=中文男" \
  -F "instruct_text=Speak with courage and determination" \
  --output output.wav
```

### Response
#### Success Response (200)
- Content-Type: `audio/wav`
- Body: Streaming audio with instruction-controlled synthesis

#### Response Example
(Streaming audio/wav)
```

--------------------------------

### Import Main CosyVoice Classes

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/INDEX.md

Import the factory function and different versions of the base CosyVoice model. Use AutoModel as a factory, and CosyVoice, CosyVoice2, or CosyVoice3 for specific model implementations.

```python
from cosyvoice.cli.cosyvoice import (
    AutoModel,
    CosyVoice,
    CosyVoice2,
    CosyVoice3
)
```

--------------------------------

### CosyVoice Constructor Parameters

Source: https://github.com/funaudiollm/cosyvoice/blob/main/_autodocs/configuration.md

Use this to initialize the base CosyVoice model. Optimization flags like load_jit, load_trt, and fp16 require CUDA.

```python
CosyVoice(
    model_dir: str,
    load_jit: bool = False,
    load_trt: bool = False,
    fp16: bool = False,
    trt_concurrent: int = 1
)
```