### Install UniVoice Dependencies

Source: https://github.com/gwh22/univoice/blob/main/README.md

Commands to clone the repository and set up the required Python environment with CUDA support.

```shell
git clone https://github.com/gwh22/UniVoice
cd UniVoice
# We recommend using conda to create a new environment.
conda create -n UniVoice python=3.10
conda activate UniVoice
# install cuda >= 11.8
conda install cudatoolkit=11.8 -c nvidia

pip install -r requirements.txt
```

--------------------------------

### Configure Training Pipeline

Source: https://context7.com/gwh22/univoice/llms.txt

Executes a distributed training job for ASR and TTS capabilities using torchrun with mixed precision.

```bash
#!/bin/bash
export PYTHONPATH=.
export NCCL_P2P_DISABLE=1

# Training configuration
torchrun --nproc_per_node=4 --master_port=25901 univoice/train/train_scratch.py \
    --model_name_or_path /path/to/SmolLM2-360M-Instruct \
    --tts_train_data_path data/libriheavy_train \
    --asr_train_data_path data/libriheavy_train \
    --eval_data_path data/librispeech_eval \
    --task all \
    --bf16 True \
    --output_dir checkpoints/univoice \
    --epochs 10 \
    --max_steps 1000000 \
    --batch_size_per_gpu 20000 \
    --max_samples 64 \
    --batch_size_type frame \
    --n_mel_channels 80 \
    --target_sample_rate 22050 \
    --gradient_accumulation_steps 1 \
    --lr 1e-3 \
    --mixed_precision bf16 \
    --max_grad_norm 1 \
    --weight_decay 0.05 \
    --warmup_iters 100 \
    --lr_decay_iters 500000 \
    --lr_decay_rate 0.1 \
    --save_per_updates 10000 \
    --log_steps 100 \
    --logger tensorboard
```

--------------------------------

### Train UniVoice Model

Source: https://github.com/gwh22/univoice/blob/main/README.md

Execute the training script for the UniVoice model.

```shell
cd UniVoice
sh scripts/train_all.sh
```

--------------------------------

### Load UniVoiceForCausalLM Model and Tokenizer

Source: https://context7.com/gwh22/univoice/llms.txt

Loads the main UniVoice model and tokenizer, sets up special tokens for speech processing, and loads checkpoint weights. Ensure the paths to the LLM and checkpoint are correct. The model is set to evaluation mode and moved to CUDA.

```python
import torch
from transformers import AutoTokenizer
from univoice.model import UniVoiceForCausalLM
from univoice.constants import DEFAULT_SPEECH_TOKEN, DEFAULT_PAD_TOKEN, DEFAULT_SPEECH_START_TOKEN, DEFAULT_SPEECH_END_TOKEN
from univoice.train.train_utils import setup_tokenizer

# Load model and tokenizer
llm_path = "/path/to/SmolLM2-360M"
ckpt_path = "/path/to/univoice_checkpoint"

tokenizer = AutoTokenizer.from_pretrained(llm_path)
model = UniVoiceForCausalLM.from_pretrained(llm_path, torch_dtype=torch.float32)

# Setup special tokens for speech processing
model, tokenizer = setup_tokenizer(model, tokenizer)
model.config.speech_token_index = tokenizer.convert_tokens_to_ids(DEFAULT_SPEECH_TOKEN)
model.config.eos_token_id = tokenizer.eos_token_id
model.config.bos_token_id = tokenizer.bos_token_id
model.config.pad_token_id = tokenizer.pad_token_id

# Load checkpoint weights
checkpoint = torch.load(f'{ckpt_path}/model_last.pt', map_location='cuda')
model.load_state_dict(checkpoint["model_state_dict"], strict=False)
model.eval().cuda()

print(f"Model loaded with {model.parameter_count()} parameters")
```

--------------------------------

### Euler Sampling for Flow Matching

Source: https://context7.com/gwh22/univoice/llms.txt

Performs Euler sampling for flow matching-based speech generation, utilizing classifier-free guidance and sway sampling. This method is typically called internally by model.sample().

```python
import torch

# euler_sample is called internally by model.sample()
# Key parameters:
# - sample_N: number of sampling steps (default: 32)
# - guidance_scale: CFG scale for generation quality
# - sway_sampling_coef: -1.0 for sway sampling schedule

# The method implements:
# 1. Uniform timestep schedule with sway sampling adjustment
# 2. Forward passes for conditional and unconditional predictions
# 3. Classifier-free guidance: pred = cond + (cond - uncond) * scale
# 4. CFG-renorm to prevent magnitude explosion
# 5. Euler step: x_t+1 = x_t + pred * dt

# Example usage (internal to model.sample):
mel_output, num_function_evaluations = model.euler_sample(
    input_ids=input_ids,
    attention_mask=attention_mask,
    labels=labels,
    speechs=reference_mel,
    mel_spec=noise_tensor,      # Initial random noise
    step_cond=conditioning_mel,  # Reference audio conditioning
    flags=torch.tensor([0]),     # TTS flag
    target_len=target_length,
    guidance_scale=2.0
)
```

--------------------------------

### Initialize Whisper Projection Layer

Source: https://context7.com/gwh22/univoice/llms.txt

Maps Whisper encoder outputs to the LLM hidden dimension for ASR tasks.

```python
import torch
from univoice.model.builder import WhisperProjection

# Initialize projection layer
# Maps Whisper encoder output (1280-dim) to LLM hidden size (960-dim for SmolLM2)
projector = WhisperProjection(
    input_embedding_size=1280,  # Whisper-large hidden size
    output_embedding_size=960   # SmolLM2-360M hidden size
)

# Example: project Whisper encoder output
```

--------------------------------

### Create Padding Mask

Source: https://context7.com/gwh22/univoice/llms.txt

Generates a padding mask given sequence lengths and a maximum length. `True` values in the output mask indicate padded positions.

```python
import torch
from univoice.util.utils_smollm import make_pad_mask

# Create padding mask
lengths = torch.tensor([10, 15, 8, 20])
pad_mask = make_pad_mask(lengths, max_len=25)
print(f"Pad mask shape: {pad_mask.shape}")  # (4, 25)
# True values indicate padded positions
```

--------------------------------

### Run UniVoice Inference

Source: https://github.com/gwh22/univoice/blob/main/README.md

Execute ASR or TTS inference tasks using the provided shell scripts.

```shell
cd UniVoice
# for ASR task
sh scripts/infer_asr.sh
# for TTS task
sh scripts/infer_tts.sh
```

--------------------------------

### Create Random Span Mask for Infilling Training

Source: https://context7.com/gwh22/univoice/llms.txt

Generates a random span mask for infilling training, similar to F5-TTS. It masks a fraction of each sequence, determined by `frac_lengths`, within a batch.

```python
import torch
from univoice.util.utils_smollm import mask_from_frac_lengths

# Create random span mask for infilling training
# Used in F5-TTS style training where random spans are masked
batch_size = 4
seq_lengths = torch.tensor([100, 120, 90, 110])
frac_lengths = torch.zeros(batch_size).uniform_(0.7, 1.0)  # Mask 70-100% of sequence
span_mask = mask_from_frac_lengths(seq_lengths, frac_lengths)
print(f"Span mask shape: {span_mask.shape}")  # (4, 120)
```

--------------------------------

### Initialize and Encode Timesteps with TimestepEmbedder

Source: https://context7.com/gwh22/univoice/llms.txt

Initializes the TimestepEmbedder with specified hidden size and frequency embedding size. Encodes random timesteps scaled to the 0-1000 range, outputting embeddings with shape (batch_size, hidden_size).

```python
import torch
from univoice.model.univoice_smollm import TimestepEmbedder

# Initialize timestep embedder
t_embedder = TimestepEmbedder(
    hidden_size=960,
    frequency_embedding_size=256
)

# Encode timesteps (scaled to 0-1000 range)
batch_size = 4
timesteps = torch.rand(batch_size) * 1000  # Random timesteps

# Output shape: (batch_size, hidden_size)
t_embeds = t_embedder(timesteps.cuda())
print(f"Timestep embeddings shape: {t_embeds.shape}")
# Output: torch.Size([4, 960])
```

--------------------------------

### Cite UniVoice

Source: https://github.com/gwh22/univoice/blob/main/README.md

BibTeX citation for the UniVoice research paper.

```bibtex
@article{guan2025univoice,
  title={UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models},
  author={Guan, Wenhao and Niu, Zhikang and Jiang, Ziyue and Wang, Kaidi and Chen, Peijie and Hong, Qingyang and Li, Lin and Chen, Xie},
  journal={arXiv preprint arXiv:2510.04593},
  year={2025}
}
```

--------------------------------

### Perform TTS Inference with Voice Cloning

Source: https://context7.com/gwh22/univoice/llms.txt

Generates speech from text using flow matching and Euler sampling, supporting zero-shot voice cloning via reference audio.

```python
import torch
import torchaudio
from univoice.model import UniVoiceForCausalLM
from univoice.util.utils_smollm import MelSpec_bigvGAN, preprocess_single_inputs

# Load reference audio for voice cloning (22.05kHz for TTS)
ref_wav_path = "/path/to/reference_voice.flac"
audio, source_sample_rate = torchaudio.load(ref_wav_path)

if audio.shape[0] > 1:
    audio = torch.mean(audio, dim=0, keepdim=True)

if source_sample_rate != 22050:
    resampler = torchaudio.transforms.Resample(source_sample_rate, 22050)
    audio = resampler(audio)

# Extract mel spectrogram using BigVGAN-compatible extractor
mel_spec = MelSpec_bigvGAN(
    n_fft=1024,
    hop_length=256,
    win_length=1024,
    n_mel_channels=80,
    target_sample_rate=22050,
)(audio).to('cuda')

# Text to synthesize
text = ["hello world, this is a test of the univoice text to speech system"]

# Set target duration in seconds (affects output length)
duration_seconds = 5.0
target_len = torch.tensor([int(duration_seconds * 22050 // 256)])

# Generate speech using flow matching
with torch.no_grad():
    mel_out, mel_gt = model.sample(
        input_ids=torch.randn(1).to('cuda'),
        attention_mask=torch.randn(1).to('cuda'),
        labels=torch.randn(1).to('cuda'),
        mel_spec=mel_spec.transpose(1,2),
        speechs=mel_spec,
        flags=torch.tensor([0]),  # flag=0 for TTS
        target_len=target_len,
        text=text,
        cfg_scale=2.0  # Classifier-free guidance scale
    )

# Convert mel spectrogram to waveform using BigVGAN vocoder
from BigVGAN import bigvgan
vocoder = bigvgan.BigVGAN.from_pretrained('bigvgan_22k', use_cuda_kernel=False)
vocoder.remove_weight_norm()
vocoder = vocoder.eval().to('cuda')

with torch.inference_mode():
    wav_gen = vocoder(mel_out.squeeze(0).transpose(0,1).unsqueeze(0))

# Save output audio
wav_gen_float = wav_gen.squeeze(0).cpu()
torchaudio.save('output.wav', wav_gen_float, 22050)
```

--------------------------------

### Extract Mel Spectrograms for BigVGAN

Source: https://context7.com/gwh22/univoice/llms.txt

Computes mel spectrograms compatible with the BigVGAN vocoder using librosa-based filterbanks.

```python
import torch
import torchaudio
from univoice.util.utils_smollm import MelSpec_bigvGAN

# Initialize mel spectrogram extractor for TTS
mel_extractor = MelSpec_bigvGAN(
    n_fft=1024,
    hop_length=256,
    win_length=1024,
    n_mel_channels=80,
    target_sample_rate=22050,
    fmin=0,
    fmax=8000,
    center=False
)

# Load and process audio
audio, sr = torchaudio.load("input.wav")
if sr != 22050:
    audio = torchaudio.transforms.Resample(sr, 22050)(audio)

# Extract mel spectrogram
# Input shape: (batch, samples) or (batch, 1, samples)
# Output shape: (batch, n_mel_channels, time_frames)
mel_spec = mel_extractor(audio)
print(f"Mel spectrogram shape: {mel_spec.shape}")
# Example output: torch.Size([1, 80, 431]) for ~5 seconds of audio
```

--------------------------------

### Special Tokens for UniVoice

Source: https://context7.com/gwh22/univoice/llms.txt

Defines special tokens used in UniVoice for tasks like Automatic Speech Recognition (ASR) and Text-to-Speech (TTS), including tokens for loss masking, padding, and speech boundaries.

```python
from univoice.constants import (
    IGNORE_INDEX,
    DEFAULT_PAD_TOKEN,
    DEFAULT_SPEECH_TOKEN,
    DEFAULT_SPEECH_START_TOKEN,
    DEFAULT_SPEECH_END_TOKEN,
)

# Example prompt construction for ASR
asr_prompt = f"{tokenizer.bos_token}{DEFAULT_SPEECH_TOKEN}<|asr|>")
# Result: "<|im_start|><SPEECH><|asr|>")

# Example prompt construction for TTS
tts_prompt = f"<|im_start|>{text}{DEFAULT_SPEECH_START_TOKEN}{DEFAULT_SPEECH_TOKEN}{DEFAULT_SPEECH_END_TOKEN}")
# Result: "<|im_start|>Hello world<|speech_start|><SPEECH><|speech_end|>")
```

--------------------------------

### ASR Inference - Speech to Text

Source: https://context7.com/gwh22/univoice/llms.txt

Performs automatic speech recognition. This snippet loads an audio file, preprocesses it into mel spectrograms using WhisperFeatureExtractor, and then uses the UniVoice model with flag=1 for ASR task to generate a transcription. Ensure the audio is 16kHz or resampled to it.

```python
import torch
import torchaudio
from transformers import WhisperFeatureExtractor
from univoice.model import UniVoiceForCausalLM
from univoice.constants import DEFAULT_SPEECH_TOKEN

# Load and preprocess audio (expects 16kHz for ASR)
wav_path = "/path/to/audio.flac"
audio, source_sample_rate = torchaudio.load(wav_path)

# Convert to mono if needed
if audio.shape[0] > 1:
    audio = torch.mean(audio, dim=0, keepdim=True)

# Resample to 16kHz for Whisper
if source_sample_rate != 16000:
    resampler = torchaudio.transforms.Resample(source_sample_rate, 16000)
    audio = resampler(audio)

# Extract mel spectrogram using Whisper feature extractor
feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-large-v3-turbo")
mel_spec = feature_extractor(audio.numpy(), sampling_rate=16000).input_features[0]
mel_spec = torch.tensor(mel_spec, dtype=torch.float32)  # Shape: (80, 3000)

# Prepare inputs for ASR (flag=1 indicates ASR task)
speechs = [mel_spec.to('cuda')]
inputs = [f"{tokenizer.bos_token}{DEFAULT_SPEECH_TOKEN}<|asr|>"]

input_ids, attention_mask = tokenizer(
    inputs, max_length=512, truncation=True,
    add_special_tokens=False, return_tensors="pt"
).values()

# Generate transcription
with torch.inference_mode():
    output_ids = model.generate(
        input_ids=input_ids.to('cuda'),
        attention_mask=attention_mask.to('cuda'),
        speechs=speechs[0].transpose(0,1).unsqueeze(0),
        flags=torch.tensor([1], dtype=torch.int32),  # flag=1 for ASR
        t=torch.tensor([0]).to('cuda'),
        temperature=0.7,
        top_p=0.95,
        top_k=50,
        num_beams=4,
        repetition_penalty=1.2,
        eos_token_id=tokenizer.eos_token_id,
    )

transcription = tokenizer.batch_decode(output_ids, skip_special_tokens=False)[0].strip()
print(f"Transcription: {transcription}")
```

--------------------------------

### Create Attention Mask from Sequence Lengths

Source: https://context7.com/gwh22/univoice/llms.txt

Generates an attention mask from a tensor of sequence lengths. The mask is boolean, with `True` values indicating positions that should be attended to.

```python
import torch
from univoice.util.utils_smollm import lens_to_mask

# Create attention mask from sequence lengths
lengths = torch.tensor([10, 15, 8, 20])
mask = lens_to_mask(lengths, length=20)
print(f"Attention mask shape: {mask.shape}")  # (4, 20)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.