Step-Audio

https://github.com/stepfun-ai/step-audio
Admin
Step-Audio is the first production-ready open-source framework for intelligent speech interaction...
Tokens:68,982
Snippets:399
Trust Score:8.3
License:Apache-2.0
Update:1 month ago
Show doc for...
Tokens:
Raw
### AutoFrontend usage example

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/auto-frontend.md

Shows how to instantiate AutoFrontend with a model, frontend type, and configuration, then call it on an audio file to get feature batches.

```python
from funasr_detach import AutoFrontend

frontend = AutoFrontend(
    model="iic/speech_paraformer-large_asr_nat-zh-cantonese-en-16k-vocab8501-online",
    frontend="default",
    frontend_conf={"fs": 16000, "n_fft": 512},
)

batches = frontend("speech.wav", fs=16000)
feat, feat_len, keys = batches[0]["input"], batches[0]["input_len"], batches[0]["key"]
```

--------------------------------

### Install dependencies and clone Step-Audio models

Source: https://github.com/stepfun-ai/step-audio/blob/main/README.md

Install dependencies and clone the Step-Audio repository and all three model repositories (Step-Audio-Tokenizer, Step-Audio-Chat, Step-Audio-TTS-3B) from Hugging Face. Requires Python >= 3.10, PyTorch >= 2.3-cu121, and CUDA Toolkit.

```bash
git clone https://github.com/stepfun-ai/Step-Audio.git
conda create -n stepaudio python=3.10
conda activate stepaudio

cd Step-Audio
pip install -r requirements.txt

git lfs install
git clone https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer
git clone https://huggingface.co/stepfun-ai/Step-Audio-Chat
git clone https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B

```

--------------------------------

### Usage Examples

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Examples of using StepAudioTTS for plain TTS with emotion/language/rate tags, voice cloning with a reference wav, and error handling for unknown speakers.

```APIDOC
## Usage

### Plain TTS with emotion, language and rate tags

```python
import torchaudio
from tts import StepAudioTTS
from tokenizer import StepAudioTokenizer

encoder = StepAudioTokenizer("/models/Step-Audio-Tokenizer")
tts = StepAudioTTS("/models/Step-Audio-TTS-3B", encoder)

text = "(高兴1)(中文)(快速1)今天天气真不错，我们出去玩吧！"
audio, sr = tts(text, "Tingting")   # sr == 22050
torchaudio.save("happy_fast.wav", audio, sr)
```

### Voice cloning with a reference wav and its transcript

```python
import torchaudio
from tts import StepAudioTTS
from tokenizer import StepAudioTokenizer

encoder = StepAudioTokenizer("/models/Step-Audio-Tokenizer")
tts = StepAudioTTS("/models/Step-Audio-TTS-3B", encoder)

clone_speaker = {
    "speaker": "custom_voice",
    "prompt_text": "这秋风一起啊，螃蟹就该上市了。",
    "wav_path": "examples/prompt_wav_yuqian.wav",
}
audio, sr = tts("人活一辈子，有高峰，有低谷。", "", clone_speaker)
torchaudio.save("clone.wav", audio, sr)
```

### Error handling — unknown speaker without clone raises a `TypeError` (unsubscriptable `None`), unknown RAP speaker raises `KeyError`; catch both:

```python
try:
    audio, sr = tts("（RAP）随便说唱两句", "Nobody")
except (KeyError, TypeError) as exc:
    print(f"speaker lookup failed: {exc}")
```
```

--------------------------------

### CosyVoiceModel usage example

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Shows how to instantiate CosyVoiceModel and run inference: load flow.pt and hift.pt, then call flow.inference to get mel and hift.inference to get waveform.

```python
from cosyvoice.cli.model import CosyVoiceModel

model = CosyVoiceModel(flow_module, hift_module)
model.load("/models/.../flow.pt", "/models/.../hift.pt")
mel = model.flow.inference(token=tokens, token_len=..., prompt_token=..., prompt_token_len=...,
                           prompt_feat=feat, prompt_feat_len=feat_len, embedding=emb)
wav = model.hift.inference(mel=mel.float())[0].cpu()
```

--------------------------------

### Basic text-in/audio-out dialogue with StepAudio

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio.md

Shows how to initialize StepAudio with tokenizer, TTS, and LLM paths, then call the model with a user message to get text, audio, and sample rate. The audio is saved to a WAV file and the text response is printed.

```python
import torchaudio
from stepaudio import StepAudio

model = StepAudio(
    tokenizer_path="/models/Step-Audio-Tokenizer",
    tts_path="/models/Step-Audio-TTS-3B",
    llm_path="/models/Step-Audio-Chat",
)

text, audio, sr = model(
    [{"role": "user", "content": "你好，我是你的朋友，我叫小明，你叫什么名字？"}],
    "Tingting",
)
torchaudio.save("reply.wav", audio, sr)
print(text)
```

--------------------------------

### CosyVoiceModel Usage Example

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Example showing how to instantiate CosyVoiceModel, load the models, and perform inference using the flow and hift modules.

```APIDOC
## CosyVoiceModel Usage Example

### Description
Example showing how to instantiate CosyVoiceModel, load the models, and perform inference using the flow and hift modules.

### Code
```python
from cosyvoice.cli.model import CosyVoiceModel

model = CosyVoiceModel(flow_module, hift_module)
model.load("/models/.../flow.pt", "/models/.../hift.pt")
mel = model.flow.inference(token=tokens, token_len=..., prompt_token=..., prompt_token_len=...,
                           prompt_feat=feat, prompt_feat_len=feat_len, embedding=emb)
wav = model.hift.inference(mel=mel.float())[0].cpu()
```
```

--------------------------------

### Basic encode with StepAudioTokenizer

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tokenizer.md

Basic encoding example used by StepAudio.encode_audio. Loads an audio file and encodes it to a token string using the StepAudioTokenizer.

```python
import torchaudio
from tokenizer import StepAudioTokenizer
from utils import load_audio

encoder = StepAudioTokenizer("/models/Step-Audio-Tokenizer")

wav, sr = load_audio("question.wav")
token_str = encoder(wav, sr)          # -> "<audio_65536><audio_65537>..."
print(token_str[:80])
```

--------------------------------

### Usage of frontend extractors

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice-frontend.md

Usage example showing how to load a CosyVoice model, access its frontend, and call the three extractor methods with appropriately resampled audio.

```python
import torchaudio
from cosyvoice.cli.cosyvoice import CosyVoice

cosy = CosyVoice("/models/Step-Audio-TTS-3B/CosyVoice-300M-25Hz")
frontend = cosy.frontend

wav_22k, sr = torchaudio.load("prompt.wav")            # 22050 Hz
wav_16k, _ = torchaudio.load("prompt.wav")             # 16000 Hz

speech_feat, speech_feat_len = frontend._extract_speech_feat(wav_22k)
embedding = frontend._extract_spk_embedding(wav_16k)
token, token_len = frontend._extract_speech_token(wav_16k)
```

--------------------------------

### POST /v1/chat/completions

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/endpoints.md

This snippet documents the OpenAI-compatible chat completions endpoint exposed by the vLLM server for the Step-Audio-Chat model. It includes the request schema, example, and response structure.

```APIDOC
## POST /v1/chat/completions

### Description
Sends a chat completion request to the Step-Audio-Chat model served via vLLM. This endpoint is compatible with the OpenAI chat completions API and returns the model's text reply.

### Method
POST

### Endpoint
{server_url}/v1/chat/completions

### Parameters
#### Headers
- **Authorization** (string) - Required - Any non-empty value is accepted. No authentication is enforced.
- **Content-Type** (string) - Required - Must be `application/json`.

#### Request Body
- **model** (string) - Required - The served model name, as specified by the `--served-model-name` argument when starting the vLLM server.
- **messages** (array) - Required - A standard OpenAI chat message list. Each message must have a `role` and `content`. Audio input is not supported on this route.

### Request Example
{
  "model": "step-audio-chat",
  "messages": [
    {"role": "system", "content": "You are an AI designed for conversation, currently unable to connect to the internet."},
    {"role": "user", "content": "Introduce yourself."}
  ]
}

### Response
#### Success Response (200)
- **choices[0].message.content** (string) - The model's text reply.

#### Response Example
{
  "choices": [
    {
      "message": {
        "content": "Hello! I am an AI assistant..."
      }
    }
  ]
}

### Error Handling
- **404** - Unknown model name.
- **400** - Malformed request body.
- **200** - Successful completion.

No authentication is required.
```

--------------------------------

### AudioTokenString token string example

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/types.md

Example of an `AudioTokenString` produced by `StepAudioTokenizer.__call__` / `merge_vq0206_to_token_str` and consumed by `utils.get_audio_tokens`. Semantic ids occupy 65536..65536+1023; acoustic ids start at 65536+1024. The LLM emits these tokens directly; `StepAudioTTS` subtracts 65536 before the CosyVoice decoder.

```text
"<audio_65536><audio_65537><audio_66560>..."
```

--------------------------------

### Error handling for token_to_wav_offline

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Error handling example for token_to_wav_offline. Catches AssertionError which occurs when batch size is not 1 or mel lengths are inconsistent.

```python
try:
    wav = cosy.token_to_wav_offline(bad_token_beam, feat, feat_len, pt, pt_len, emb)
except AssertionError:
    print("flow decoder requires batch size 1 and consistent mel lengths")
```

--------------------------------

### Handle 30s speech-token limit

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice-frontend.md

Error handling example for the 30-second speech-token limit, which is raised as an AssertionError.

```python
try:
    token, token_len = frontend._extract_speech_token(long_wav)
except AssertionError:
    print("audio exceeds the 30s speech-token limit; split it first")
```

--------------------------------

### Voice cloning with reference wav and transcript

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Example of voice cloning using a reference WAV file and its transcript. The clone_speaker dictionary specifies the speaker name, prompt text, and wav path. The generated audio is saved as clone.wav.

```python
import torchaudio
from tts import StepAudioTTS
from tokenizer import StepAudioTokenizer

encoder = StepAudioTokenizer("/models/Step-Audio-Tokenizer")
tts = StepAudioTTS("/models/Step-Audio-TTS-3B", encoder)

clone_speaker = {
    "speaker": "custom_voice",
    "prompt_text": "这秋风一起啊，螃蟹就该上市了。",
    "wav_path": "examples/prompt_wav_yuqian.wav",
}
audio, sr = tts("人活一辈子，有高峰，有低谷。", "", clone_speaker)
torchaudio.save("clone.wav", audio, sr)
```

--------------------------------

### Direct call to token_to_wav_offline

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Direct usage example of token_to_wav_offline. Shows required tensor shapes and dtypes. Note that speech_feat, prompt_token, and embedding normally come from the frontend and encoder; this is a sketch of the call shape.

```python
import torch
from cosyvoice.cli.cosyvoice import CosyVoice
from cosyvoice.cli.frontend import CosyVoiceFrontEnd

cosy = CosyVoice("/models/Step-Audio-TTS-3B/CosyVoice-300M-25Hz")

# speech_feat / prompt_token / embedding come from the frontend plus an encoder;
# here we only sketch the call shape used by StepAudioTTS:
wav = cosy.token_to_wav_offline(
    speech_token=torch.tensor([[1, 2, 3]]),   # [1, T] ids (offset already removed)
    speech_feat=torch.zeros(1, 100, 80, dtype=torch.bfloat16),
    speech_feat_len=torch.tensor([100], dtype=torch.int32),
    prompt_token=torch.tensor([[1]], dtype=torch.long),
    prompt_token_len=torch.tensor([1], dtype=torch.long),
    embedding=torch.zeros(1, 192, dtype=torch.bfloat16),
)
```

--------------------------------

### Handle AssertionError from get_vq06_code

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tokenizer.md

Error handling example for decoder token-rate mismatch. Catches AssertionError from get_vq06_code when acoustic token count deviates from 25 Hz rate by more than 2 tokens.

```python
try:
    codes = encoder.get_vq06_code(wav)
except AssertionError:
    codes = None
    # acoustic token count deviated from 25 Hz rate by > 2 tokens
```

--------------------------------

### Launch web demos for Chat and TTS

Source: https://github.com/stepfun-ai/step-audio/blob/main/README.md

Launch local web demos for Step-Audio-Chat and Step-Audio-TTS-3B. Assumes 4 GPUs are available and all models are downloaded.

```bash
# Step-Audio-Chat demo
python app.py --model-path where_you_download_dir

# Step-Audio-TTS-3B demo
python tts_app.py --model-path where_you_download_dir
```

--------------------------------

### Build and run Step-Audio Docker containers

Source: https://github.com/stepfun-ai/step-audio/blob/main/README.md

Build and run the Step-Audio Docker images. The first set builds and runs the standard image, the second builds and runs the vLLM image with an additional port 8000 exposed.

```bash
# build docker
docker build . -t step-audio

# run docker
docker run --rm -ti --gpus all \
    -v /your/code/path:/app -v /your/model/path:/model \
    -p 7860:7860 \
    step-audio \
    -- bash

# build vLLM docker
docker build -f Dockerfile-vllm -t step-audio-vllm .

# run vLLM docker
docker run --rm -ti --gpus all \
    -v /your/code/path:/app -v /your/model/path:/model \
    -p 7860:7860 \
    -p 8000:8000 \
    step-audio-vllm \
    -- bash
```

--------------------------------

### StepAudioTokenizer.__init__

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tokenizer.md

Initializes the StepAudioTokenizer, loading the FunASR model, k-means centroids for semantic quantization, and the ONNX acoustic tokenizer.

```APIDOC
## `StepAudioTokenizer` — Constructor

### Description
Initializes the StepAudioTokenizer with the provided encoder path, resolving the FunASR model, k-means centroids, and ONNX speech tokenizer artifacts.

### Method
`__init__`

### Signature
```python
def __init__(
    self,
    encoder_path,
)
```

### Parameters
- `encoder_path` (str) - Required - Model directory containing the FunASR model directory, `linguistic_tokenizer.npy`, and `speech_tokenizer_v1.onnx`.

### Errors
- **AssertionError** - If the nested FunASR model directory is missing.
- **FileNotFoundError** - If `linguistic_tokenizer.npy` or `speech_tokenizer_v1.onnx` is missing.
- **onnxruntime errors** - If CUDA provider is unavailable.

```

--------------------------------

### `load_optimus_ths_lib(libpath)`

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/utils.md

Loads a prebuilt flash-attention library, selecting the appropriate file based on the torch version. Returns a boolean indicating success.

```APIDOC
## `load_optimus_ths_lib(libpath)`

### Description
Loads a prebuilt flash-attention library for the Optimus THS integration. The library file is selected based on the installed PyTorch version. This function never raises errors; it returns a boolean indicating success and prints a failure message if loading fails.

### Method
Function call

### Endpoint
`utils.load_optimus_ths_lib`

### Parameters
- `libpath` (str) - Required - Directory containing the prebuilt `liboptimus_ths-*.so` flash-attention libraries.

### Return
`bool` — `True` when a library was successfully loaded (cached on the function object as `load_optimus_ths_lib.success`), `False` otherwise.

### Behavior
- Loading is guarded by a module-level `threading.Lock`.
- A success flag short-circuits repeat loads.
- The chosen `.so` file depends on `torch.__version__`:
  - `>= 2.5`: `liboptimus_ths-torch2.5-cu124.cpython-310-x86_64-linux-gnu.so`
  - `>= 2.3`: `liboptimus_ths-torch2.3-cu121.cpython-310-x86_64-linux-gnu.so`
  - `>= 2.2`: `liboptimus_ths-torch2.2-cu121.cpython-310-x86_64-linux-gnu.so`
  - else: `raise RuntimeError("Unsupported torch version")` (caught and swallowed).

### Errors
- Any failure prints `Fail to load optimus_ths and flash attn is disabled: {err}` and returns `False`. The exception never escapes this function.
- The source comment notes that `LD_LIBRARY_PATH` must contain `nvidia/cuda_nvrtc/lib` (typically `.../site-packages/nvidia/cuda_nvrtc/lib`) for the CUDA runtime.
```

--------------------------------

### StepAudio.__init__

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio.md

Initializes the StepAudio object by loading the tokenizer, TTS model, and LLM. It loads the Optimus THS library, initializes the LLM tokenizer, encoder, decoder, and the 130B chat LLM.

```APIDOC
## StepAudio.__init__

### Description
Initializes the StepAudio object by loading the tokenizer, TTS model, and LLM. It loads the Optimus THS library, initializes the LLM tokenizer, encoder, decoder, and the 130B chat LLM.

### Method
Constructor

### Signature
```python
def __init__(self, tokenizer_path: str, tts_path: str, llm_path: str):
```

### Parameters
- **tokenizer_path** (str) - Required - Directory containing the Step-Audio-Tokenizer model (holds the FunASR Paraformer dir, `linguistic_tokenizer.npy` and `speech_tokenizer_v1.onnx`).
- **tts_path** (str) - Required - Directory containing the Step-Audio-TTS-3B model (holds `CosyVoice-300M-25Hz`, `CosyVoice-300M-25Hz-Music`, the TTS LLM and `lib/`).
- **llm_path** (str) - Required - Directory containing the Step-Audio-Chat 130B LLM and its `lib/` flash-attention library.

### Behavior
1. `load_optimus_ths_lib(os.path.join(llm_path, 'lib'))` — loads the custom Optimus THS flash-attention shared library; failure is non-fatal.
2. `self.llm_tokenizer = AutoTokenizer.from_pretrained(llm_path, trust_remote_code=True)`.
3. `self.encoder = StepAudioTokenizer(tokenizer_path)`.
4. `self.decoder = StepAudioTTS(tts_path, self.encoder)`.
5. `self.llm = AutoModelForCausalLM.from_pretrained(llm_path, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)` — the 130B chat LLM is loaded in bfloat16 with automatic device mapping (typically needs 4×80 GB GPUs).

### Errors
| Error type | Condition |
|------------|-----------|
| `TypeError` | Wrong keyword names, e.g. the stale `encoder_path=`/`decoder_path=` used in the module's own `__main__` block. |
| downstream exceptions | Any failure inside the three constructors (missing model files, missing `lib/`, OOM during `device_map="auto"`). |
```

--------------------------------

### Error handling for missing model kwarg

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/auto-model.md

Shows that AutoModel() without the required 'model' kwarg raises an AssertionError. This is a minimal example to demonstrate the mandatory argument.

```python
try:
    AutoModel()          # missing required 'model' kwarg
except AssertionError:
    print("AutoModel requires model=...")
```

--------------------------------

### Serve Step-Audio-Chat with vLLM

Source: https://github.com/stepfun-ai/step-audio/blob/main/README.md

Serve Step-Audio-Chat with vLLM using tensor parallelism. Must export the custom flash attention library path (OPTIMUS_LIB_PATH) before running. The model does not support audio input because vLLM does not load Tokenizer and TTS.

```bash
export OPTIMUS_LIB_PATH=where_you_download_dir/Step-Audio-Chat/lib

vllm serve where_you_download_dir/Step-Audio-Chat --dtype auto -tp $tp --served-model-name step-audio-chat --trust-remote-code

# vLLM chat example code
python call_vllm_chat.py
```

--------------------------------

### AutoModel.__init__

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/auto-model.md

Initializes the AutoModel with model configuration, device selection, and optional VAD, punctuation, and speaker models.

```APIDOC
## AutoModel.__init__

### Description
Initializes the AutoModel inference facade with the specified model and configuration. It builds the main model and optionally VAD, punctuation, and speaker models.

### Method
Constructor

### Signature
```python
def __init__(self, **kwargs):
```

### Parameters
- **model** (str) - Required - Model name or local path, resolved through name_maps_ms/name_maps_hf and downloaded when missing.
- **model_revision** (str) - Optional - Model hub revision.
- **model_hub** (str) - Optional - "ms" (ModelScope) or "hf" (Hugging Face). Default: "ms".
- **model_conf** (dict) - Optional - Direct model config; when absent the config is downloaded/merged from the hub.
- **device** (str) - Optional - Device to use, defaults to "cuda" with fallback to "cpu" when CUDA unavailable or ngpu == 0.
- **ngpu** (int) - Optional - Number of GPUs; 0 forces CPU. Default: 1.
- **ncpu** (int) - Optional - When set, calls torch.set_num_threads(ncpu).
- **seed** (int) - Optional - Random seed. Default: 0.
- **tokenizer** (str) - Optional - Registered tokenizer class name.
- **tokenizer_conf** (dict) - Optional - Tokenizer config.
- **frontend** (str) - Optional - Registered frontend class name.
- **frontend_conf** (dict) - Optional - Frontend config.
- **init_param** (str) - Optional - Pretrained checkpoint path.
- **vad_model** (str) - Optional - VAD model name; enables inference_with_vad.
- **vad_model_revision** (str) - Optional - VAD revision.
- **punc_model** (str) - Optional - Punctuation model; applied after ASR in VAD mode.
- **punc_model_revision** (str) - Optional - Punctuation revision.
- **spk_model** (str) - Optional - Speaker model for diarization; requires a punc_model.
- **spk_model_revision** (str) - Optional - Speaker revision.
- **spk_mode** (str) - Optional - One of "default", "vad_segment", "punc_segment". Default: "punc_segment".
- **disable_log** (bool) - Optional - Suppresses registry table print. Default: False.
- **batch_size** (int) - Optional - Inference batch size. Default: 1.
- **disable_pbar** (bool) - Optional - Disables tqdm progress bars. Default: False.

### Errors
- **AssertionError** - Raised if "model" is not provided in kwargs.

### Example
```python
from funasr_detach import AutoModel
model = AutoModel(model="my_model", device="cuda")
```
```

--------------------------------

### Manual two-codebook path with wav2token and get_vq02_code

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tokenizer.md

Manual two-codebook path used internally by StepAudioTTS.preprocess_prompt_wav. Shows wav2token for semantic tokens and get_vq02_code for streaming encode with session cache.

```python
import torchaudio
from tokenizer import StepAudioTokenizer

encoder = StepAudioTokenizer("/models/Step-Audio-Tokenizer")
wav, sr = torchaudio.load("prompt.wav", backend="soundfile")

speech_tokens, vq02, vq06 = encoder.wav2token(wav, sr)
# semantic-only cluster ids for prompt registration:
prompt_code, _, _ = encoder.wav2token(wav, sr)

# streaming encode with session cache is not directly supported by wav2token;
# it is exposed via get_vq02_code(audio, session_id=..., is_final=False)
chunk = encoder.preprocess_wav(wav, sr)
codes = encoder.get_vq02_code(chunk, session_id="s1", is_final=False)
codes += encoder.get_vq02_code(chunk, session_id="s1", is_final=True)
```

--------------------------------

### Error handling for unknown speakers

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Example of catching TypeError and KeyError when using an unknown speaker without cloning or an unknown RAP speaker. Prints an error message with the exception.

```python
try:
    audio, sr = tts("（RAP）随便说唱两句", "Nobody")
except (KeyError, TypeError) as exc:
    print(f"speaker lookup failed: {exc}")
```

--------------------------------

### Set OPTIMUS_LIB_PATH for Step-Audio-Chat

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/configuration.md

Set the OPTIMUS_LIB_PATH environment variable to point vLLM at the custom flash-attention library for Step-Audio-Chat. Must be set before running `vllm serve`.

```bash
export OPTIMUS_LIB_PATH=dir/Step-Audio-Chat/lib
```

--------------------------------

### Expected model directory structure

Source: https://github.com/stepfun-ai/step-audio/blob/main/README.md

Expected directory structure after downloading the models. The where_you_download_dir should contain the three model folders.

```text
where_you_download_dir
├── Step-Audio-Tokenizer
├── Step-Audio-Chat
├── Step-Audio-TTS-3B
```

--------------------------------

### StepAudioTTS.__init__

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Initializes the StepAudioTTS instance by loading the TTS model, tokenizer, CosyVoice models, and registering speakers.

```APIDOC
## StepAudioTTS.__init__

### Description
Initializes the StepAudioTTS instance. Loads the TTS LLM checkpoint, tokenizer, CosyVoice models, and registers speakers.

### Method
Constructor

### Signature
```python
def __init__(self, model_path, encoder):
```

### Parameters
- **model_path** (str) - Required - Path to the Step-Audio-TTS-3B directory containing the TTS LLM checkpoint, `CosyVoice-300M-25Hz/`, `CosyVoice-300M-25Hz-Music/`, and `lib/`.
- **encoder** (StepAudioTokenizer) - Required - Shared tokenizer instance used to tokenize prompt/reference audio.

### State Created
- Loads `lib` via `load_optimus_ths_lib`.
- Loads LLM model with `AutoModelForCausalLM.from_pretrained`.
- Loads tokenizer with `AutoTokenizer.from_pretrained`.
- Loads common CosyVoice model from `CosyVoice-300M-25Hz`.
- Loads music CosyVoice model from `CosyVoice-300M-25Hz-Music`.
- Registers speakers via `register_speakers()`.

### System Prompts
- `sys_prompt_for_rap`: Instructs the model to rap the text using the voice from dialogue history.
- `sys_prompt_for_vocal`: Instructs the model to hum/sing the text.
- `sys_prompt_wo_spk`: Voice-actor prompt listing supported emotion, language/dialect, music, and speech-rate tags.
- `sys_prompt_with_spk`: Same as `wo_spk` but adds "使用[{}]的声音" with the prompt speaker.

### Errors
- `FileNotFoundError`: Missing `speakers/speakers_info.json` or any `speakers/{speaker_id}_prompt.wav`.
- Downstream exceptions: Missing CosyVoice model dirs, missing `lib/`, OOM.
```

--------------------------------

### Run call_vllm_chat.py to query vLLM-served chat model

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/scripts.md

Calls the Step-Audio-Chat model via a vLLM OpenAI-compatible server. --server-url defaults to http://127.0.0.1:8000 and --model-name to step-audio-chat. Prints the assistant's reply.

```bash
python call_vllm_chat.py --server-url http://127.0.0.1:8000 --model-name step-audio-chat
```

--------------------------------

### Import StepAudio

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio.md

Import the StepAudio class from the stepaudio module.

```python
from stepaudio import StepAudio
```

--------------------------------

### Plain TTS usage with emotion, language, and rate tags

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Example of using StepAudioTTS for text-to-speech with emotion, language, and rate tags in the input text. The output audio is saved as a WAV file with sample rate 22050 Hz.

```python
import torchaudio
from tts import StepAudioTTS
from tokenizer import StepAudioTokenizer

encoder = StepAudioTokenizer("/models/Step-Audio-Tokenizer")
tts = StepAudioTTS("/models/Step-Audio-TTS-3B", encoder)

text = "(高兴1)(中文)(快速1)今天天气真不错，我们出去玩吧！"
audio, sr = tts(text, "Tingting")   # sr == 22050
torchaudio.save("happy_fast.wav", audio, sr)
```

--------------------------------

### AutoFrontend

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/auto-frontend.md

Documentation for the AutoFrontend class, including constructor and __call__ method.

```APIDOC
## AutoFrontend

### Description
Builds a registered speech frontend (STFT → WPE → beamformer → mel-Fbank → CMVN pipeline) and slices raw audio into feature batches.

### Constructor

```python
def __init__(self, **kwargs):
```

#### Parameters
- **model** (str) - Required - Model name used to download the frontend config.
- **model_hub** (str) - Optional - Model hub selector (default: "ms").
- **frontend** (str) - Optional - Registered frontend class name, e.g. "default", "fused", "s3prl".
- **frontend_conf** (dict) - Optional - Frontend configuration (fs defaults, n_fft, etc.).

### __call__

```python
def __call__(self, input, input_len=None, kwargs=None, **cfg):
```

#### Parameters
- **input** (str/list/bytes) - Required - Audio paths / urls / raw data.
- **input_len** - Optional - Input lengths.
- **kwargs** (dict) - Optional - Kwargs override (defaults to self.kwargs).
- **cfg** - Optional - Runtime overrides: batch_size (default 1), fs (default 16000), data_type (default "sound").

#### Returns
- **list[dict]** - One batch dict per chunk, each containing:
  - **input** (torch.Tensor) - Extracted fbank features [B, T, dim].
  - **input_len** (torch.Tensor) - Frame lengths.
  - **key** (list[str]) - Batch keys.

### Usage Example

```python
from funasr_detach import AutoFrontend

frontend = AutoFrontend(
    model="iic/speech_paraformer-large_asr_nat-zh-cantonese-en-16k-vocab8501-online",
    frontend="default",
    frontend_conf={"fs": 16000, "n_fft": 512},
)

batches = frontend("speech.wav", fs=16000)
feat, feat_len, keys = batches[0]["input"], batches[0]["input_len"], batches[0]["key"]
```
```

--------------------------------

### Import CosyVoiceFrontEnd

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice-frontend.md

Import the CosyVoiceFrontEnd class from the cosyvoice.cli.frontend module.

```python
from cosyvoice.cli.frontend import CosyVoiceFrontEnd
```

--------------------------------

### Import StepAudioTokenizer

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tokenizer.md

Import the StepAudioTokenizer class from the tokenizer module. The class is the main entry point for converting raw audio into discrete speech tokens.

```python
from tokenizer import StepAudioTokenizer
```

--------------------------------

### Import AutoFrontend and AutoTokenizer

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/auto-frontend.md

Shows the import paths for AutoFrontend and AutoTokenizer from the funasr_detach package and its auto submodule.

```python
from funasr_detach.auto.auto_frontend import AutoFrontend
from funasr_detach import AutoFrontend  # package re-export

from funasr_detach.auto.auto_tokenizer import AutoTokenizer
```

--------------------------------

### load_optimus_ths_lib function signature

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/utils.md

Signature of the load_optimus_ths_lib function. It takes a directory containing the prebuilt liboptimus_ths-*.so flash-attention libraries and returns a bool indicating whether a library was successfully loaded (cached on the function object as load_optimus_ths_lib.success).

```python
def load_optimus_ths_lib(libpath):
```

--------------------------------

### AutoFrontend constructor signature

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/auto-frontend.md

Defines the __init__ method of AutoFrontend. It accepts model, model_hub, frontend, and frontend_conf parameters, and asserts that 'model' is present.

```python
def __init__(self, **kwargs):
```

--------------------------------

### Run tts_inference.py for TTS or voice-clone inference

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/scripts.md

Runs TTS or voice-clone inference. --model-path must contain Step-Audio-Tokenizer/ and Step-Audio-TTS-3B/. --output-path is required and created if missing. --synthesis-type defaults to "tts"; any other value triggers clone mode.

```bash
python tts_inference.py --model-path where_you_download_dir \
    --output-path where_you_save_audio_dir --synthesis-type use_tts_or_clone
```

--------------------------------

### tokenize

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Builds the full LLM input token sequence for text-to-speech generation, combining system prompt, prompt audio tokens, and target text into a chat-formatted token list.

```APIDOC
## `tokenize`

### Description
Builds the full LLM input token sequence for text-to-speech generation, combining system prompt, prompt audio tokens, and target text into a chat-formatted token list.

### Method
`tokenize(self, text: str, prompt_text: str, prompt_speaker: str, prompt_code: list)`

### Parameters
- **text** (`str`) - Required - Target text.
- **prompt_text** (`str`) - Required - Prompt (speaker or clone) transcript.
- **prompt_speaker** (`str`) - Required - Prompt speaker id (`''` when cloning).
- **prompt_code** (`list`) - Required - Prompt audio dual-codebook token ids (semantic + acoustic, `+65536`-offset).

### Return
`list[int]` — full LLM input token sequence.

### Format Built (`tts.py:153-199`)
`system\n{prompt}` wrapped in `[4] ... [3]`; then a `human -> assistant` prompt-audio exchange (with `prompt_code` as the assistant's audio tokens) followed by `human -> target_text` and a final `assistant` role; `[4]`/`[3]` are the chat delimiter ids, `[1]`/`[4]`/`[3]` bracket the whole conversation. System prompt selection: `RAP`/`哼唱` → rap/vocal prompt; non-empty `prompt_speaker` → `sys_prompt_with_spk.format(prompt_speaker)`; else `sys_prompt_wo_spk`.
```

--------------------------------

### StepAudioTTS register_speakers method

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Loads speaker information from speakers_info.json and preprocesses prompt wav files for each speaker. Prints a message for each registered speaker.

```python
def register_speakers(self):
```

--------------------------------

### `load_audio(audio_path)`

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/utils.md

Loads an audio file and returns a mono waveform and sample rate.

```APIDOC
## `load_audio(audio_path)``

### Description
Loads an audio file into a mono waveform and its sample rate. The audio file must be readable by `torchaudio.load`.

### Method
Function call

### Endpoint
`utils.load_audio`

### Parameters
- `audio_path` (str) - Required - Path to an audio file readable by `torchaudio.load`.

### Return
2-tuple `(audio_wav, sr)` — mono waveform (`mean(dim=0, keepdim=True)`) and its sample rate.
```

--------------------------------

### Run offline_inference.py for end-to-end chat inference

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/scripts.md

Runs end-to-end chat inference. Requires --model-path to point to a base directory containing Step-Audio-Tokenizer/, Step-Audio-TTS-3B/, and Step-Audio-Chat/ subdirectories. Produces output/output_e2e_tqta.wav and output/output_e2e_aqta.wav.

```bash
python offline_inference.py --model-path where_you_download_dir
```

--------------------------------

### CosyVoice.__init__

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Initializes the CosyVoice decoder by loading configuration, speaker embedding model, acoustic tokenizer, flow-matching decoder, and HiFT vocoder from the specified model directory.

```APIDOC
## CosyVoice.__init__

### Description
Initializes the CosyVoice decoder by loading the model configuration, speaker embedding model, acoustic tokenizer, flow-matching decoder, and HiFT vocoder from the specified model directory.

### Method
Constructor

### Signature
```python
def __init__(
    self,
    model_dir,
):
```

### Parameters
- **model_dir** (str) - Required - CosyVoice model directory. Must contain the following files:
  - `cosyvoice.yaml` - hyperpyyaml config defining `feat_extractor`, `flow`, `hift` sections
  - `campplus.onnx` - speaker-embedding model
  - `speech_tokenizer_v1.onnx` - acoustic tokenizer
  - `flow.pt` - flow-matching decoder state dict
  - `hift.pt` - HiFT vocoder state dict

### Errors
- **FileNotFoundError** - If any of the required files are missing.
- **omegaconf errors** - If `cosyvoice.yaml` is missing `feat_extractor`, `flow`, or `hift` keys.

### Notes
- Creates `self.frontend = CosyVoiceFrontEnd(feat_extractor, campplus.onnx, speech_tokenizer_v1.onnx)`
- Creates `self.model = CosyVoiceModel(configs["flow"], configs["hift"])`
- Loads model weights and sets flow decoder to bfloat16; vocoder remains float32.
```

--------------------------------

### Import StepAudioTTS and RepetitionAwareLogitsProcessor

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Import the main TTS engine and its repetition-suppression processor from the tts module.

```python
from tts import StepAudioTTS, RepetitionAwareLogitsProcessor
```

--------------------------------

### Audio input with voice-speed control

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio.md

Demonstrates audio-to-text-to-audio generation with a speed_ratio of 1.2. The model accepts an audio content dict and returns text, audio, and sample rate, which are saved to a WAV file.

```python
import torchaudio
from stepaudio import StepAudio

model = StepAudio("<tokenizer>", "<tts-3b>", "<chat>")

# audio-in (audio-to-text-to-audio), sped up 1.2x
text, audio, sr = model(
    [{"role": "user", "content": {"type": "audio", "audio": "question.wav"}}],
    "Tingting",
    speed_ratio=1.2,
)
torchaudio.save("answer_fast.wav", audio, sr)
```

--------------------------------

### StepAudio constructor signature

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio.md

Constructor signature for StepAudio. Takes paths to the tokenizer, TTS, and LLM model directories.

```python
def __init__(self, tokenizer_path: str, tts_path: str, llm_path: str):
```

--------------------------------

### Shared audio loading and post-processing pattern

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/utils.md

Shared loading pattern used by every entry point. It loads an audio file, applies energy normalization, trims silence, and post-processes a synthesized waveform with speed and volume adjustments before encoding to base64.

```python
import torchaudio
import utils

wav, sr = utils.load_audio("prompt.wav")
wav = utils.energy_norm_fn(wav)
wav = utils.trim_silence(wav.numpy().squeeze(0), 16000)
wav = torch.from_numpy(wav).unsqueeze(0)

# post-process a synthesized waveform
audio = utils.speech_adjust(audio, sr, speed_ratio=1.5)
audio = utils.volumn_adjust(audio, sr, volumn_ratio=1.2)
b64 = utils.encode_wav(audio, sr)
```

--------------------------------

### StepAudioTTS constructor

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Initializes the StepAudioTTS instance with a model path and tokenizer. Loads the TTS LLM, CosyVoice models, and registers speakers. Throws FileNotFoundError if speaker files are missing.

```python
def __init__(
    self,
    model_path,
    encoder,
):
```

--------------------------------

### CosyVoiceFrontEnd constructor

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice-frontend.md

Constructor signature for CosyVoiceFrontEnd. It takes a mel-feature callable, the path to campplus.onnx, and the path to speech_tokenizer_v1.onnx.

```python
def __init__(
    self,
    feat_extractor: Callable,
    campplus_model: str,
    speech_tokenizer_model: str,
):
```

--------------------------------

### CosyVoiceModel.__init__

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Constructor for the CosyVoiceModel class. Initializes the model with a flow-matching mel decoder and a HiFT vocoder, and sets the device to CUDA if available, otherwise CPU.

```APIDOC
## CosyVoiceModel.__init__

### Description
Constructor for the CosyVoiceModel class. Initializes the model with a flow-matching mel decoder and a HiFT vocoder, and sets the device to CUDA if available, otherwise CPU.

### Method
Constructor

### Signature
```python
def __init__(
    self,
    flow: torch.nn.Module,
    hift: torch.nn.Module,
):
```

### Parameters
- **flow** (torch.nn.Module) - Required - Mel decoder module (from `configs["flow"]` in `cosyvoice.yaml`).
- **hift** (torch.nn.Module) - Required - Waveform vocoder module (from `configs["hift"]`).

### State
- `self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")`
```

--------------------------------

### Import AutoModel and prepare_data_iterator

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/auto-model.md

Import the AutoModel class and prepare_data_iterator from the funasr_detach.auto.auto_model module. The package root also re-exports AutoModel.

```python
from funasr_detach.auto.auto_model import AutoModel, prepare_data_iterator
# re-exported from the package root:
from funasr_detach import AutoModel
```

--------------------------------

### StepAudioTokenizer.__call__

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tokenizer.md

Converts raw audio into a full audio-token string by invoking the internal tokenization pipeline.

```APIDOC
## `StepAudioTokenizer.__call__`

### Description
Processes an audio waveform and returns the full audio-token string.

### Method
`__call__`

### Signature
```python
def __call__(self, audio, sr):
```

### Parameters
- `audio` (torch.Tensor) - Required - Waveform; shape `[1, samples]`.
- `sr` (int) - Required - Sample rate of the audio.

### Returns
- **str** - Full audio-token string, e.g., `"<audio_65536>..."`.

```

--------------------------------

### CosyVoiceModel constructor

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Initializes the model with a flow mel decoder and hift vocoder. Sets self.device to cuda if available, otherwise cpu.

```python
def __init__(
    self,
    flow: torch.nn.Module,
    hift: torch.nn.Module,
):
```

--------------------------------

### Import CosyVoice and CosyVoiceModel

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Import the CosyVoice and CosyVoiceModel classes from the cosyvoice package. Use this to access the speech decoder and model classes.

```python
from cosyvoice.cli.cosyvoice import CosyVoice
from cosyvoice.cli.model import CosyVoiceModel
```

--------------------------------

### CosyVoice constructor

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/cosyvoice.md

Constructor for the CosyVoice class. Takes a model directory containing required files: cosyvoice.yaml, campplus.onnx, speech_tokenizer_v1.onnx, flow.pt, and hift.pt. Initializes the frontend and model, and moves the flow decoder to bfloat16.

```python
def __init__(
    self,
    model_dir,
):
```

--------------------------------

### preprocess_prompt_wav method signature

Source: https://github.com/stepfun-ai/step-audio/blob/main/_autodocs/api-reference/stepaudio-tts.md

Method signature for preprocess_prompt_wav, which processes a prompt WAV file and returns a 6-tuple of tensors including prompt codes, mel features, and speaker embedding. The wav is resampled to 16000 Hz for embedding and 22050 Hz for mel features.

```python
def preprocess_prompt_wav(self, prompt_wav_path : str):
```
Tokens:
Raw