Zonos

https://github.com/zyphra/zonos
Admin
Zonos-v0.1 is a leading open-weight text-to-speech model trained on more than 200k hours of varied...
Tokens:35,684
Snippets:360
Trust Score:7.5
License:Apache-2.0
Update:2 months ago
Show doc for...
Tokens:
Raw
### Verify installation

Source: https://github.com/zyphra/zonos/blob/main/README.md

Run the provided sample script to confirm the installation is successful.

```bash
uv run sample.py
# python sample.py
```

--------------------------------

### Conditioner specification and example

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Format for individual conditioner specifications and a full example configuration.

```python
{
    "type": "EspeakPhonemeConditioner",  # or FourierConditioner, IntegerConditioner, PassthroughConditioner
    "name": "espeak",                     # Unique key for cond_dict
    "cond_dim": 512,                      # Input dimension (optional)
    "projection": "none",                 # Per-conditioner projection
    "uncond_type": "none",                # "learned" for learnable unconditional or "none"
}
```

```python
PrefixConditionerConfig(
    conditioners=[
        {"type": "EspeakPhonemeConditioner", "name": "espeak"},
        {"type": "PassthroughConditioner", "name": "speaker", "cond_dim": 256},
        {"type": "FourierConditioner", "name": "emotion", "std": 1.0, "min_val": 0.0, "max_val": 1.0},
        {"type": "FourierConditioner", "name": "fmax", "std": 1.0, "min_val": 0.0, "max_val": 24000.0},
        {"type": "FourierConditioner", "name": "pitch_std", "std": 1.0, "min_val": 0.0, "max_val": 400.0},
        {"type": "FourierConditioner", "name": "speaking_rate", "std": 1.0, "min_val": 0.0, "max_val": 40.0},
        {"type": "IntegerConditioner", "name": "language_id", "min_val": 0, "max_val": 80},
    ],
    projection="linear"
)
```

--------------------------------

### Install into active environment using uv

Source: https://github.com/zyphra/zonos/blob/main/README.md

Install project dependencies into an existing environment using uv.

```bash
uv pip install -e .
uv pip install -e .[compile] # optional but needed to run the hybrid
```

--------------------------------

### Example: Prepare conditioning

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Demonstrates creating a conditioning dictionary and preparing it for the model.

```python
from zonos.conditioning import make_cond_dict

cond_dict = make_cond_dict(text="Hello, world!", speaker=speaker_embedding)
conditioning = model.prepare_conditioning(cond_dict)
```

--------------------------------

### Launch Gradio Interface

Source: https://github.com/zyphra/zonos/blob/main/README.md

Starts the recommended Gradio web interface for repeated sampling.

```bash
uv run gradio_interface.py
# python gradio_interface.py
```

--------------------------------

### Example: Generate audio

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Demonstrates calling the generate method with specific sampling parameters.

```python
codes = model.generate(
    prefix_conditioning=conditioning,
    max_new_tokens=86 * 10,
    cfg_scale=2.0,
    sampling_params={"min_p": 0.1, "top_p": 0.95}
)
```

--------------------------------

### Install into a new uv virtual environment

Source: https://github.com/zyphra/zonos/blob/main/README.md

Recommended method for setting up the project environment using uv.

```bash
uv sync
uv sync --extra compile # optional but needed to run the hybrid
uv pip install -e .
```

--------------------------------

### Install into active environment using pip

Source: https://github.com/zyphra/zonos/blob/main/README.md

Install project dependencies into an existing environment using standard pip.

```bash
pip install -e .
pip install --no-build-isolation -e .[compile] # optional but needed to run the hybrid
```

--------------------------------

### BackboneConfig usage examples

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Example configurations for pure transformer and hybrid model architectures.

```python
BackboneConfig(
    d_model=1024,
    n_layer=24,
    attn_cfg={"num_heads": 16, "num_heads_kv": 4},
    # ssm_cfg is empty {}
)
```

```python
BackboneConfig(
    d_model=1024,
    n_layer=24,
    d_intermediate=2752,
    attn_mlp_d_intermediate=3072,
    attn_layer_idx=[12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
    ssm_cfg={"d_state": 16, "d_conv": 4},
    attn_cfg={"num_heads": 16, "num_heads_kv": 4},
    rms_norm=True,
    residual_in_fp32=True,
)
```

--------------------------------

### Load ZonosConfig from dictionary

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Method signature and usage example for loading configuration from a dictionary.

```python
@classmethod
def from_dict(cls, d: dict) -> "ZonosConfig"
```

```python
import json
from zonos.config import ZonosConfig

with open("config.json") as f:
    config_dict = json.load(f)
config = ZonosConfig.from_dict(config_dict)
```

--------------------------------

### Configuration JSON Schema

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/10-constants-types.md

Example structure for the model configuration file.

```json
{
  "backbone": {
    "d_model": 1024,
    "d_intermediate": 0,
    "attn_mlp_d_intermediate": 3072,
    "n_layer": 24,
    "ssm_cfg": {},
    "attn_layer_idx": [],
    "attn_cfg": {
      "num_heads": 16,
      "num_heads_kv": 4
    },
    "rms_norm": false,
    "residual_in_fp32": false,
    "norm_epsilon": 1e-5
  },
  "prefix_conditioner": {
    "conditioners": [
      {"type": "EspeakPhonemeConditioner", "name": "espeak"},
      {"type": "PassthroughConditioner", "name": "speaker", "cond_dim": 256},
      {"type": "FourierConditioner", "name": "emotion", "std": 1.0, "min_val": 0.0, "max_val": 1.0},
      {"type": "FourierConditioner", "name": "fmax", "std": 1.0, "min_val": 0.0, "max_val": 24000.0},
      {"type": "FourierConditioner", "name": "pitch_std", "std": 1.0, "min_val": 0.0, "max_val": 400.0},
      {"type": "FourierConditioner", "name": "speaking_rate", "std": 1.0, "min_val": 0.0, "max_val": 40.0},
      {"type": "IntegerConditioner", "name": "language_id", "min_val": 0, "max_val": 80}
    ],
    "projection": "linear"
  },
  "eos_token_id": 1024,
  "masked_token_id": 1025,
  "pad_vocab_to_multiple_of": 8
}
```

--------------------------------

### Setup Inference Cache

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Method signature for allocating KV cache and inference parameters.

```python
def setup_cache(
    self, 
    batch_size: int, 
    max_seqlen: int, 
    dtype: torch.dtype = torch.bfloat16
) -> InferenceParams
```

--------------------------------

### Install eSpeak dependency

Source: https://github.com/zyphra/zonos/blob/main/README.md

Install the required eSpeak library for phonemization on supported operating systems.

```bash
apt install -y espeak-ng # For Ubuntu
# brew install espeak-ng # For MacOS
```

--------------------------------

### Text-to-Speech Generation Example

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/02-conditioning-api.md

Demonstrates loading a model, creating a speaker embedding from reference audio, and generating speech with custom conditioning.

```python
import torch
import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict
from zonos.utils import DEFAULT_DEVICE

model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device=DEFAULT_DEVICE)

# Load reference audio for voice cloning
wav, sr = torchaudio.load("reference.wav")
speaker = model.make_speaker_embedding(wav, sr)

# Create conditioning for happy, slow speech
cond_dict = make_cond_dict(
    text="Hello, world!",
    language="en-us",
    speaker=speaker,
    emotion=[0.8, 0, 0, 0, 0, 0, 0.1, 0.1],  # 80% happiness
    pitch_std=80.0,  # Expressive
    speaking_rate=10.0,  # Slow
    fmax=22050.0
)

conditioning = model.prepare_conditioning(cond_dict)
codes = model.generate(conditioning)
wav = model.autoencoder.decode(codes)
torchaudio.save("output.wav", wav[0], 44100)
```

--------------------------------

### Install and run Zonos with Docker

Source: https://github.com/zyphra/zonos/blob/main/README.md

Commands to clone the repository and launch the application using Docker Compose or manual build and run steps.

```bash
git clone https://github.com/Zyphra/Zonos.git
cd Zonos

# For gradio
docker compose up

# Or for development you can do
docker build -t zonos .
docker run -it --gpus=all --net=host -v /path/to/Zonos:/Zonos -t zonos
cd /Zonos
python sample.py # this will generate a sample.wav in /Zonos
```

--------------------------------

### Install Hybrid Model Dependencies

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Required packages for running the hybrid model variant.

```bash
uv sync --extra compile
# or
pip install flash-attn mamba-ssm causal-conv1d
```

--------------------------------

### Load Local Model

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Method signature and usage example for loading a model from the local filesystem.

```python
@classmethod
def from_local(
    cls, 
    config_path: str, 
    model_path: str, 
    device: str = DEFAULT_DEVICE, 
    backbone: str | None = None
) -> "Zonos"
```

```python
model = Zonos.from_local(
    config_path="/path/to/config.json",
    model_path="/path/to/model.safetensors",
    device="cuda"
)
```

--------------------------------

### Configure sampling parameters for generation

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/04-sampling-api.md

Examples demonstrating different sampling configurations to balance voice quality and creativity, including the use of the NovelAI Unified Sampler.

```python
# Conservative, high-quality audio (recommended for TTS)
codes = model.generate(
    prefix_conditioning=conditioning,
    sampling_params={
        "temperature": 1.0,
        "min_p": 0.1,
        "top_p": 0.95,
        "repetition_penalty": 3.0,
    }
)

# Creative, more varied speech
codes = model.generate(
    prefix_conditioning=conditioning,
    sampling_params={
        "temperature": 1.2,
        "top_p": 0.9,
        "min_p": 0.05,
        "repetition_penalty": 1.5,
    }
)

# NovelAI Unified Sampler for balanced outputs
codes = model.generate(
    prefix_conditioning=conditioning,
    sampling_params={
        "temperature": 1.0,
        "linear": 0.5,
        "conf": 0.1,  # -quad / 2
        "quad": -0.2,
        "repetition_penalty": 2.5,
    }
)
```

--------------------------------

### Apply Delay Pattern Example

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/07-utility-types.md

Demonstrates applying the delay pattern to a tensor of codes, resulting in an expanded sequence length.

```python
import torch
from zonos.codebook_pattern import apply_delay_pattern

# Codes from model.generate()
codes = torch.randint(0, 1024, (1, 9, 100))  # [batch=1, codebooks=9, seq_len=100]

# Apply delay pattern
delayed = apply_delay_pattern(codes, mask_token=1025)
print(delayed.shape)  # [1, 9, 109] - expanded by 9 tokens
```

--------------------------------

### Load Pretrained Model

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Method signature and usage example for loading a model from the Hugging Face Hub.

```python
@classmethod
def from_pretrained(
    cls, 
    repo_id: str, 
    revision: str | None = None, 
    device: str = DEFAULT_DEVICE, 
    **kwargs
) -> "Zonos"
```

```python
import torch
from zonos.model import Zonos
from zonos.utils import DEFAULT_DEVICE

model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device=DEFAULT_DEVICE)
```

--------------------------------

### Revert Delay Pattern Example

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/07-utility-types.md

Demonstrates the round-trip process of applying and then reverting the delay pattern to recover the original tensor shape.

```python
from zonos.codebook_pattern import apply_delay_pattern, revert_delay_pattern

codes = torch.randint(0, 1024, (1, 9, 100))
delayed = apply_delay_pattern(codes, mask_token=1025)
recovered = revert_delay_pattern(delayed)

assert recovered.shape == codes.shape  # [1, 9, 100]
```

--------------------------------

### Example Usage of SpeakerEmbeddingLDA

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

Demonstrates loading an audio file and generating speaker embeddings using the LDA embedder.

```python
import torchaudio
from zonos.speaker_cloning import SpeakerEmbeddingLDA

embedder = SpeakerEmbeddingLDA(device="cuda")
wav, sr = torchaudio.load("reference.wav")
raw_emb, lda_emb = embedder(wav, sr)
print(f"LDA embedding shape: {lda_emb.shape}")  # [1, 128]
```

--------------------------------

### Generate Speaker Embedding

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Method signature and usage example for creating a speaker embedding from an audio file for voice cloning.

```python
def make_speaker_embedding(self, wav: torch.Tensor, sr: int) -> torch.Tensor
```

```python
import torchaudio
wav, sr = torchaudio.load("reference_voice.mp3")
speaker_embedding = model.make_speaker_embedding(wav, sr)
```

--------------------------------

### Zonos.setup_cache

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Allocates KV cache and inference parameters for efficient generation.

```APIDOC
## Zonos.setup_cache

### Description
Allocates KV cache and inference parameters for efficient generation.

### Parameters
- **batch_size** (int) - Required - Batch size for inference
- **max_seqlen** (int) - Required - Maximum sequence length
- **dtype** (torch.dtype) - Optional - Data type for cache allocation

### Returns
- **InferenceParams** - Object with allocated KV cache dictionaries and tracking tensors.
```

--------------------------------

### Load Zonos from local files

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/08-usage-guide.md

Initialize the model using local configuration and weight files. Specify the backbone as either 'torch' or 'mamba_ssm'.

```python
from zonos.model import Zonos
from zonos.utils import DEFAULT_DEVICE

# Load from pretrained safetensors
model = Zonos.from_local(
    config_path="config.json",
    model_path="model.safetensors",
    device=DEFAULT_DEVICE,
    backbone="torch"  # or "mamba_ssm" for hybrid
)

# Use as normal
from zonos.conditioning import make_cond_dict

cond_dict = make_cond_dict(text="Custom model test")
conditioning = model.prepare_conditioning(cond_dict)
codes = model.generate(conditioning)
```

--------------------------------

### Get symbol IDs from text

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/07-utility-types.md

Converts a phoneme string into a list of integer token IDs.

```python
def get_symbol_ids(text: str) -> list[int]
```

--------------------------------

### Configure and Initialize Zonos Model

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Shows how to load configuration from a JSON file or define model architecture and conditioners programmatically before initializing the Zonos model.

```python
import json
import torch
from zonos.model import Zonos
from zonos.config import ZonosConfig, BackboneConfig, PrefixConditionerConfig

# Load from JSON
with open("config.json") as f:
    config_dict = json.load(f)

config = ZonosConfig.from_dict(config_dict)

# Or construct programmatically
backbone_config = BackboneConfig(
    d_model=1024,
    n_layer=24,
    attn_cfg={"num_heads": 16, "num_heads_kv": 4},
)

prefix_config = PrefixConditionerConfig(
    conditioners=[
        {"type": "EspeakPhonemeConditioner", "name": "espeak"},
        {"type": "PassthroughConditioner", "name": "speaker", "cond_dim": 256},
        {"type": "FourierConditioner", "name": "emotion"},
        {"type": "IntegerConditioner", "name": "language_id"},
    ],
    projection="linear"
)

config = ZonosConfig(
    backbone=backbone_config,
    prefix_conditioner=prefix_config,
    eos_token_id=1024,
    masked_token_id=1025,
)

# Create model
model = Zonos(config).to("cuda", torch.bfloat16)
```

--------------------------------

### MambaSSMZonosBackbone.__init__

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/06-backbone-api.md

Initializes the hybrid backbone with the provided configuration.

```APIDOC
## Constructor

### Description
Initializes the MambaSSMZonosBackbone with a specific configuration, setting up the mixed SSM and attention layers.

### Signature
`__init__(self, config: BackboneConfig)`

### Parameters
- **config** (BackboneConfig) - Required - Configuration object specifying layer indices and SSM/attention configurations.
```

--------------------------------

### Complete Speaker Cloning Workflow

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

Demonstrates loading the model, extracting speaker embeddings, preparing conditioning, and generating audio output.

```python
import torch
import torchaudio
from zonos.model import Zonos
from zonos.speaker_cloning import SpeakerEmbeddingLDA
from zonos.conditioning import make_cond_dict
from zonos.utils import DEFAULT_DEVICE

# Load model
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device=DEFAULT_DEVICE)

# Extract speaker embedding from reference audio
embedder = SpeakerEmbeddingLDA(device=DEFAULT_DEVICE)
ref_wav, ref_sr = torchaudio.load("reference_speaker.wav")
raw_emb, lda_emb = embedder(ref_wav, ref_sr)

# Use Zonos built-in method (recommended)
speaker_emb = model.make_speaker_embedding(ref_wav, ref_sr)

# Create conditioning with speaker embedding
cond_dict = make_cond_dict(
    text="This is a demo of voice cloning.",
    language="en-us",
    speaker=speaker_emb,
    emotion=[0.5, 0, 0, 0, 0, 0, 0, 0.5],  # Neutral
    pitch_std=30.0,
    speaking_rate=15.0
)

# Generate speech
conditioning = model.prepare_conditioning(cond_dict)
codes = model.generate(conditioning, max_new_tokens=86*5)
output_wav = model.autoencoder.decode(codes)

# Save output
torchaudio.save("cloned_voice.wav", output_wav[0].cpu(), 44100)
```

--------------------------------

### Initialize TorchZonosBackbone

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/06-backbone-api.md

Constructor for the transformer backbone. Requires a BackboneConfig with an empty ssm_cfg.

```python
def __init__(self, config: BackboneConfig)
```

--------------------------------

### Batch Processing Audio Samples

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/08-usage-guide.md

Demonstrates sequential generation of multiple audio samples using a list of conditioning dictionaries.

```python
import torch
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict

model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer")

texts = [
    "Hello, world!",
    "How are you today?",
    "The weather is nice.",
]

# Create conditioning for batch
cond_dicts = [make_cond_dict(text=text, language="en-us") for text in texts]

# Note: make_cond_dict returns individual dicts; need to stack for true batching
# This shows sequential generation (simple approach)
wavs = []
for cond_dict in cond_dicts:
    conditioning = model.prepare_conditioning(cond_dict)
    codes = model.generate(conditioning, batch_size=1)
    wav = model.autoencoder.decode(codes)
    wavs.append(wav[0].cpu())

# Save all
for i, wav in enumerate(wavs):
    import torchaudio
    torchaudio.save(f"output_{i}.wav", wav, 44100)
```

--------------------------------

### Zonos.from_local

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Loads a Zonos model instance from local filesystem paths.

```APIDOC
## Zonos.from_local

### Description
Loads a Zonos model from local configuration and weight files.

### Parameters
- **config_path** (str) - Required - Path to config.json file
- **model_path** (str) - Required - Path to model.safetensors file
- **device** (str) - Optional - Device to load model on
- **backbone** (str) - Optional - Backbone type ("torch" or "mamba_ssm")

### Returns
- **Zonos** - A Zonos instance with loaded weights.
```

--------------------------------

### DACAutoencoder.__init__

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Initializes the DACAutoencoder by loading the 'descript/dac_44khz' model from the Hugging Face Hub.

```APIDOC
## DACAutoencoder.__init__

### Description
Initializes the autoencoder instance. It automatically downloads and loads the 'descript/dac_44khz' model, sets it to evaluation mode, and disables gradient calculations.

### Signature
`def __init__(self)`
```

--------------------------------

### Zonos.from_pretrained

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Loads a Zonos model instance from the Hugging Face Hub.

```APIDOC
## Zonos.from_pretrained

### Description
Loads a pre-trained Zonos model from the Hugging Face Hub.

### Parameters
- **repo_id** (str) - Required - Repository ID (e.g., "Zyphra/Zonos-v0.1-transformer")
- **revision** (str) - Optional - Git revision (branch/tag/commit)
- **device** (str) - Optional - Device to load model on ("cuda", "cpu", etc.)

### Returns
- **Zonos** - A Zonos instance loaded with pretrained weights.
```

--------------------------------

### Initialize Zonos Class

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Constructor signature for the Zonos inference class.

```python
Zonos(config: ZonosConfig, backbone_cls=DEFAULT_BACKBONE_CLS)
```

--------------------------------

### InferenceParams Usage Pattern

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/07-utility-types.md

Demonstrates allocating cache, initializing KV caches, and updating offsets during a generation loop.

```python
import torch
from zonos.config import InferenceParams

# Allocate cache for batch_size=2, max_seqlen=5000
inference_params = InferenceParams(
    max_seqlen=5000,
    max_batch_size=2,
)

# Initialize KV caches
inference_params.key_value_memory_dict = model.backbone.allocate_inference_cache(
    batch_size=2,
    max_seqlen=5000,
    dtype=torch.bfloat16
)
inference_params.lengths_per_sample = torch.full((2,), 0, dtype=torch.int32)

# During generation:
for step in range(num_steps):
    # ... decode token ...
    inference_params.seqlen_offset += 1
    inference_params.lengths_per_sample[:] += 1
```

--------------------------------

### Initialize Zonos Model

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/06-backbone-api.md

Load a Zonos model from local paths, triggering the automatic backbone selection logic.

```python
model = Zonos.from_local(config_path, model_path, device="cuda")
```

--------------------------------

### Perform Basic Speech Synthesis

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/README.md

Initializes the Zonos model and generates audio from text. Requires a pre-trained model and appropriate device configuration.

```python
import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict
from zonos.utils import DEFAULT_DEVICE

model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device=DEFAULT_DEVICE)

cond_dict = make_cond_dict(text="Hello, world!", language="en-us")
conditioning = model.prepare_conditioning(cond_dict)
codes = model.generate(conditioning)
wav = model.autoencoder.decode(codes)

torchaudio.save("output.wav", wav[0].cpu(), 44100)
```

--------------------------------

### Check Supported Architectures

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/06-backbone-api.md

Retrieve the list of architectures supported by the backbone.

```python
MambaSSMZonosBackbone.supported_architectures
# Returns: ["transformer", "hybrid"]
```

--------------------------------

### ResNet293_based Constructor

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

Initializes the ResNet293 architecture with Attentive Statistics Pooling.

```python
def __init__(
    self,
    in_planes: int = 64,
    embd_dim: int = 256,
    acoustic_dim: int = 80,
    featCal=None,
    dropout: float = 0,
    **kwargs,
)
```

--------------------------------

### Check Supported Architectures

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/06-backbone-api.md

Retrieves the list of supported architectures for the backbone.

```python
TorchZonosBackbone.supported_architectures
# Returns: ["transformer"]
```

--------------------------------

### SpeakerEmbedding prepare_input Method

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

Resamples and normalizes audio input to 16 kHz mono format for the model.

```python
def prepare_input(self, wav: torch.Tensor, sample_rate: int) -> torch.Tensor
```

--------------------------------

### Sampling Presets Configuration

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/04-sampling-api.md

Various parameter configurations for controlling generation behavior, ranging from high-fidelity to deterministic outputs.

```python
sampling_params = {
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 0,
    "min_p": 0.1,
    "repetition_penalty": 3.0,
    "repetition_penalty_window": 2,
}
```

```python
sampling_params = {
    "temperature": 1.0,
    "top_p": 0.9,
    "min_p": 0.05,
    "repetition_penalty": 2.5,
}
```

```python
sampling_params = {
    "temperature": 1.1,
    "top_p": 0.85,
    "min_p": 0.02,
    "repetition_penalty": 2.0,
}
```

```python
sampling_params = {
    "temperature": 0,  # Argmax sampling
}
```

--------------------------------

### Preprocess Audio

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Resamples audio to 44.1 kHz and pads it to a multiple of 512 samples.

```python
def preprocess(self, wav: torch.Tensor, sr: int) -> torch.Tensor
```

```python
wav, sr = torchaudio.load("audio.wav")  # sr might be 16000
wav_preprocessed = autoencoder.preprocess(wav, sr)  # Now 44100 Hz, padded
```

--------------------------------

### Optimize Zonos Generation Performance

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/08-usage-guide.md

Demonstrates techniques to speed up generation, including using the transformer model, deterministic sampling, and limiting token output.

```python
import torch
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict

# Use transformer for speed
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device="cuda")

# Use bfloat16 (automatic)
print(f"Model dtype: {next(model.parameters()).dtype}")  # torch.bfloat16

# Batch generation if possible
cond_dict = make_cond_dict(text="Short text")
conditioning = model.prepare_conditioning(cond_dict)

# Shorter generation
codes = model.generate(
    conditioning,
    max_new_tokens=86 * 5,  # 5 seconds instead of 30
    progress_bar=False
)

# Check if CUDA Graphs available (hybrid only)
print(f"CUDA Graphs: {model.can_use_cudagraphs()}")

# Use conservative sampling
codes = model.generate(
    conditioning,
    sampling_params={
        "temperature": 0,  # Fastest: deterministic
        "repetition_penalty": 1.0  # No penalty
    }
)

wav = model.autoencoder.decode(codes)
import torchaudio
torchaudio.save("output_fast.wav", wav[0].cpu(), 44100)
```

--------------------------------

### Project Module Structure

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/README.md

Overview of the Zonos repository file organization.

```text
zonos/
├── model.py                      # Zonos class (main entry point)
├── conditioning.py               # Conditioners, phonemization, make_cond_dict()
├── speaker_cloning.py            # Speaker embeddings (ResNet293, LDA)
├── sampling.py                   # Sampling algorithms (top-p, min-p, etc.)
├── autoencoder.py                # DACAutoencoder wrapper
├── config.py                     # Dataclasses (ZonosConfig, BackboneConfig, etc.)
├── utils.py                      # Utility functions (find_multiple, get_device)
├── codebook_pattern.py           # Delay patterns for multi-codebook generation
└── backbone/
    ├── __init__.py               # Backbone registry
    ├── _torch.py                 # TorchZonosBackbone (pure PyTorch)
    └── _mamba_ssm.py             # MambaSSMZonosBackbone (hybrid)
```

--------------------------------

### prepare_conditioning

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Prepares conditioning embeddings for classifier-free guidance (CFG).

```APIDOC
## prepare_conditioning(cond_dict, uncond_dict=None)

### Description
Prepares conditioning embeddings for CFG (classifier-free guidance).

### Parameters
- **cond_dict** (dict) - Required - Conditioning dictionary with keys required by PrefixConditioner
- **uncond_dict** (dict) - Optional - Unconditional conditioning dictionary; if None, only uses required keys from cond_dict

### Returns
- **torch.Tensor** - Concatenated conditioning tensor [2 * batch_size, seq_len, d_model] for CFG.
```

--------------------------------

### Initialize PrefixConditioner

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/02-conditioning-api.md

Constructor for the PrefixConditioner class, which aggregates multiple conditioning streams.

```python
def __init__(self, config: PrefixConditionerConfig, output_dim: int)
```

--------------------------------

### Define Sampling Parameter Constants

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/10-constants-types.md

Constants for configuring temperature, Top-P, Min-P, and repetition penalty during generation.

```python
# Temperature
temperature_deterministic = 0.0   # Argmax sampling
temperature_normal = 1.0          # Standard
temperature_creative = 1.1        # More diverse

# Top-P (Nucleus Sampling)
top_p_conservative = 0.95         # Very selective
top_p_normal = 0.9                # Balanced
top_p_creative = 0.85             # More diverse
top_p_disabled = 0.0              # Disabled

# Min-P
min_p_conservative = 0.1          # Only top 10% of max prob
min_p_normal = 0.05               # Top 5%
min_p_creative = 0.02             # Top 2%
min_p_disabled = 0.0              # Disabled

# Repetition Penalty
repetition_penalty_off = 1.0      # No penalty
repetition_penalty_normal = 3.0   # Default
repetition_penalty_strong = 5.0   # Strong diversity
```

--------------------------------

### Configure Sampling Parameters for Zonos

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/08-usage-guide.md

Adjusts model output diversity using temperature, min_p, top_p, and repetition penalty settings.

```python
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict

model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer")

cond_dict = make_cond_dict(text="Hello, world!", language="en-us")
conditioning = model.prepare_conditioning(cond_dict)

# Deterministic (always same output)
codes_det = model.generate(
    conditioning,
    sampling_params={"temperature": 0}
)

# High quality, conservative
codes_hq = model.generate(
    conditioning,
    sampling_params={
        "temperature": 1.0,
        "min_p": 0.1,
        "top_p": 0.95,
        "repetition_penalty": 3.0
    }
)

# Diverse and creative
codes_diverse = model.generate(
    conditioning,
    sampling_params={
        "temperature": 1.1,
        "top_p": 0.85,
        "min_p": 0.02,
        "repetition_penalty": 1.5
    }
)

# Save all variants
for codes, name in [(codes_det, "deterministic"), 
                     (codes_hq, "high_quality"), 
                     (codes_diverse, "diverse")]:
    wav = model.autoencoder.decode(codes)
    torchaudio.save(f"output_{name}.wav", wav[0].cpu(), 44100)
```

--------------------------------

### Zonos Speaker Cloning and Generation Workflow

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

This workflow demonstrates how to load the Zonos model, extract speaker embeddings from a reference audio file, create conditioning parameters, and generate speech.

```APIDOC
## Zonos Speaker Cloning and Generation

### Description
This workflow allows users to clone a voice by extracting speaker embeddings from a reference audio file and using those embeddings to condition the Zonos model for speech synthesis.

### Usage
1. **Load Model**: Initialize the Zonos model using `Zonos.from_pretrained`.
2. **Extract Embedding**: Use `model.make_speaker_embedding(ref_wav, ref_sr)` to generate the speaker embedding from a reference audio file.
3. **Create Conditioning**: Use `make_cond_dict` to define text, language, speaker embedding, and prosody parameters (emotion, pitch, rate).
4. **Generate**: Use `model.prepare_conditioning` and `model.generate` to produce audio codes, then decode them using `model.autoencoder.decode`.

### Parameters
- **text** (str) - The text to be synthesized.
- **language** (str) - The language code (e.g., "en-us").
- **speaker** (torch.Tensor) - The speaker embedding tensor.
- **emotion** (list) - A list of floats representing emotion weights.
- **pitch_std** (float) - Standard deviation for pitch.
- **speaking_rate** (float) - The rate of speech.
```

--------------------------------

### Detecting Device with get_device

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/07-utility-types.md

Returns the preferred torch.device, prioritizing CUDA over CPU.

```python
def get_device() -> torch.device
```

--------------------------------

### Generate Speech with Zonos in Python

Source: https://github.com/zyphra/zonos/blob/main/README.md

Loads a pre-trained model, creates speaker embeddings from an audio file, and generates speech from text.

```python
import torch
import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict
from zonos.utils import DEFAULT_DEVICE as device

# model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-hybrid", device=device)
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device=device)

wav, sampling_rate = torchaudio.load("assets/exampleaudio.mp3")
speaker = model.make_speaker_embedding(wav, sampling_rate)

cond_dict = make_cond_dict(text="Hello, world!", speaker=speaker, language="en-us")
conditioning = model.prepare_conditioning(cond_dict)

codes = model.generate(conditioning)

wavs = model.autoencoder.decode(codes).cpu()
torchaudio.save("sample.wav", wavs[0], model.autoencoder.sampling_rate)
```

--------------------------------

### Load Zonos Transformer Model

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Initialize the pure transformer model variant for real-time applications.

```python
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device="cuda")
```

--------------------------------

### logFbankCal.__init__

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

Constructor for the logFbankCal class, used to initialize parameters for log mel-spectrogram feature extraction.

```APIDOC
## logFbankCal.__init__

### Description
Initializes the log mel-spectrogram feature extractor.

### Parameters
- **sample_rate** (int) - Default: 16000 - Sample rate in Hz
- **n_fft** (int) - Default: 512 - FFT size
- **win_length** (float) - Default: 0.025 - Window length in seconds
- **hop_length** (float) - Default: 0.01 - Hop length in seconds
- **n_mels** (int) - Default: 80 - Number of mel bins
```

--------------------------------

### Zonos.from_pretrained

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Loads a pre-trained Zonos model. Users can choose between the transformer-based model for speed or the hybrid model for higher quality.

```APIDOC
## Zonos.from_pretrained(model_name, device)

### Description
Loads a specific Zonos model variant from the Hugging Face hub or local path.

### Parameters
- **model_name** (str) - Required - The identifier of the model (e.g., "Zyphra/Zonos-v0.1-transformer" or "Zyphra/Zonos-v0.1-hybrid").
- **device** (str) - Optional - The compute device to load the model onto (e.g., "cuda", "cpu").

### Usage Example
```python
# Load the faster transformer model
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device="cuda")

# Load the high-quality hybrid model
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-hybrid", device="cuda")
```
```

--------------------------------

### Configure Hybrid v0.1 Backbone

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/10-constants-types.md

Defines the architecture configuration for the Hybrid v0.1 model, requiring specific GPU hardware and dependencies.

```python
BackboneConfig(
    d_model=1024,
    d_intermediate=2752,             # SSM intermediate
    attn_mlp_d_intermediate=3072,
    n_layer=24,
    ssm_cfg={
        "d_state": 16,
        "d_conv": 4,
    },
    attn_layer_idx=[12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],  # Last 12 layers
    attn_cfg={
        "num_heads": 16,
        "num_heads_kv": 4,
    },
    rms_norm=True,
    residual_in_fp32=True,
    norm_epsilon=1e-5,
)
```

--------------------------------

### Adjust Classifier-Free Guidance (CFG)

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/08-usage-guide.md

Control the strictness of conditioning adherence by adjusting the cfg_scale parameter. Higher values result in tighter adherence to the conditioning input.

```python
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict

model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer")

cond_dict = make_cond_dict(text="Test text")
conditioning = model.prepare_conditioning(cond_dict)

# Weak guidance (less strict, more natural)
codes_weak = model.generate(
    conditioning,
    cfg_scale=1.5,  # Lower = less strict
)

# Normal guidance
codes_normal = model.generate(
    conditioning,
    cfg_scale=2.0,  # Typical value
)

# Strong guidance (more strict, follows conditioning tightly)
codes_strong = model.generate(
    conditioning,
    cfg_scale=3.0,  # Higher = more strict
)

import torchaudio
for codes, name in [(codes_weak, "weak_cfg"), 
                     (codes_normal, "normal_cfg"), 
                     (codes_strong, "strong_cfg")]:
    wav = model.autoencoder.decode(codes)
    torchaudio.save(f"output_{name}.wav", wav[0].cpu(), 44100)
```

--------------------------------

### Implement generation callbacks

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/08-usage-guide.md

Use a callback function to monitor progress or trigger early stopping during generation. The callback receives the current frame, step, and maximum steps.

```python
import torch
import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict

model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer")

cond_dict = make_cond_dict(text="Hello, world!")
conditioning = model.prepare_conditioning(cond_dict)

# Callback to stop at 5 seconds
def stop_callback(frame, step, max_steps):
    max_new_tokens_5sec = 86 * 5
    return step < max_new_tokens_5sec

codes = model.generate(
    conditioning,
    max_new_tokens=86 * 30,  # Request 30 seconds
    callback=stop_callback    # But stop at 5 seconds
)

wav = model.autoencoder.decode(codes)
torchaudio.save("output_5sec.wav", wav[0].cpu(), 44100)
```

--------------------------------

### ZonosConfig.from_dict

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Loads the Zonos model configuration from a dictionary, typically parsed from a JSON file.

```APIDOC
## ZonosConfig.from_dict(d: dict)

### Description
Loads the Zonos model configuration from a dictionary. This is the standard way to initialize a ZonosConfig object from a configuration file.

### Parameters
- **d** (dict) - Required - A dictionary containing the configuration fields for the Zonos model.

### Request Example
```python
import json
from zonos.config import ZonosConfig

with open("config.json") as f:
    config_dict = json.load(f)
config = ZonosConfig.from_dict(config_dict)
```

### Response
- **Returns** (ZonosConfig) - An initialized ZonosConfig object.
```

--------------------------------

### Load Zonos Hybrid Model

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/05-autoencoder-config.md

Initialize the Mamba-attention hybrid model variant for higher quality output.

```python
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-hybrid", device="cuda")
```

--------------------------------

### Initialize Conditioner Base Class

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/02-conditioning-api.md

Constructor for the base class used by all conditioning modules.

```python
def __init__(
    self,
    output_dim: int,
    name: str,
    cond_dim: int | None = None,
    projection: Literal["none", "linear", "mlp"] = "none",
    uncond_type: Literal["learned", "none"] = "none",
    **kwargs,
)
```

--------------------------------

### Initialize SpeakerEmbeddingLDA

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

Constructor signature for the SpeakerEmbeddingLDA class.

```python
def __init__(self, device: str = DEFAULT_DEVICE)
```

--------------------------------

### Bottle2neck Constructor

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

Initializes the multi-scale residual block with branch pooling and SE module.

```python
def __init__(
    self, 
    inplanes: int, 
    planes: int, 
    kernel_size: int = None, 
    dilation: int = None, 
    scale: int = 8
)
```

--------------------------------

### Audio Prefix Continuation

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/08-usage-guide.md

Extends an existing audio file by encoding it as a prefix and generating subsequent tokens.

```python
import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict

model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer")

# Load audio prefix (e.g., from existing TTS or reference)
prefix_wav, prefix_sr = torchaudio.load("prefix.wav")

# Preprocess to 44.1 kHz
prefix_wav = model.autoencoder.preprocess(prefix_wav, prefix_sr)

# Encode prefix to codes
prefix_codes = model.autoencoder.encode(prefix_wav)

# Create conditioning for continuation
cond_dict = make_cond_dict(
    text="and then we continued with more speech.",
    language="en-us"
)

# Generate continuation
conditioning = model.prepare_conditioning(cond_dict)
codes = model.generate(
    conditioning,
    audio_prefix_codes=prefix_codes,
    max_new_tokens=86 * 10  # 10 seconds of new content
)

# Concatenate prefix and generated
combined_codes = torch.cat([prefix_codes, codes], dim=2)
combined_wav = model.autoencoder.decode(combined_codes)
torchaudio.save("output_continued.wav", combined_wav[0].cpu(), 44100)
```

--------------------------------

### SpeakerEmbeddingLDA.__init__

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/03-speaker-cloning-api.md

Constructor for initializing the speaker embedding model, which downloads pretrained weights from Hugging Face.

```APIDOC
## SpeakerEmbeddingLDA.__init__

### Description
Initializes the speaker embedding model. Downloads pretrained models from Hugging Face Hub on first instantiation.

### Signature
`def __init__(self, device: str = DEFAULT_DEVICE)`

### Parameters
- **device** (str) - Optional - Device to load models on ("cuda:0", "cpu", etc.)
```

--------------------------------

### Prepare conditioning embeddings

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/01-model-api.md

Defines the signature for preparing conditioning embeddings for classifier-free guidance.

```python
def prepare_conditioning(
    self, 
    cond_dict: dict, 
    uncond_dict: dict | None = None
) -> torch.Tensor
```

--------------------------------

### Zonos Utility Functions Usage

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/07-utility-types.md

Demonstrates sequence alignment, codebook delay pattern application, and phoneme tokenization using Zonos utilities.

```python
import torch
from zonos.utils import find_multiple, pad_weight_, DEFAULT_DEVICE
from zonos.codebook_pattern import apply_delay_pattern, revert_delay_pattern
from zonos.conditioning import tokenize_phonemes, supported_language_codes

# Device selection
device = DEFAULT_DEVICE
print(f"Using {device}")

# Alignment calculation
target_size = find_multiple(1026, 8)  # Pad embedding to 1032

# Codebook patterns
codes = torch.randint(0, 1024, (2, 9, 500))  # [batch=2, codebooks=9, time=500]
delayed_codes = apply_delay_pattern(codes, mask_token=1025)
recovered_codes = revert_delay_pattern(delayed_codes)

# Text processing
phonemes = ["h ə l oʊ", "w ɜː l d"]
token_ids, lengths = tokenize_phonemes(phonemes)

# Language validation
language = "en-us"
assert language in supported_language_codes, f"Language {language} not supported"

print(f"Processed {len(phonemes)} sequences")
print(f"Max phoneme sequence length: {max(lengths)}")
```

--------------------------------

### Generate Speech from Text

Source: https://github.com/zyphra/zonos/blob/main/_autodocs/08-usage-guide.md

Performs basic text-to-speech synthesis using default model settings and saves the output as a WAV file.

```python
import torch
import torchaudio
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict
from zonos.utils import DEFAULT_DEVICE

# Load model (downloads from Hugging Face Hub)
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device=DEFAULT_DEVICE)

# Create conditioning
cond_dict = make_cond_dict(
    text="Hello, world! This is a test of the Zonos text-to-speech system.",
    language="en-us"
)

# Generate speech
conditioning = model.prepare_conditioning(cond_dict)
codes = model.generate(conditioning)

# Decode and save
wav = model.autoencoder.decode(codes)
torchaudio.save("output.wav", wav[0].cpu(), 44100)
```
Tokens:
Raw