### Install Pre-commit Hooks

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Installs the pre-commit hooks for development. This is a standard setup step for many Python projects.

```bash
pip install pre-commit
pre-commit install
```

```bash
uvx pre-commit install
```

--------------------------------

### Install Moshi and Dependencies

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/tts_pytorch.ipynb

Installs necessary libraries for Moshi TTS. Use the first set for a fast install, or the commented-out second set for a slower but more future-proof installation that includes PyTorch and CUDA.

```python
# Fast install, might break in the future.
!pip install 'safetensors<0.6'
!pip install 'sphn<0.2'
!pip install --no-deps "moshi==0.2.11"
```

```python
# Slow install (will download torch and cuda), but future proof.
# !pip install "moshi==0.2.11"
```

--------------------------------

### Start Rust Server

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Start the moshi-server using a specified configuration file. Choose the appropriate config file based on the model you are using.

```bash
moshi-server worker --config configs/config-stt-en_fr-hf.toml
```

--------------------------------

### Start Rust TTS Server

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Command to start the Rust TTS server. Ensure 'moshi-server' is installed; a reinstall might be needed if issues arise.

```bash
moshi-server worker --config configs/config-tts.toml
```

--------------------------------

### Install Moshi Library

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/stt_pytorch.ipynb

Install the Moshi library using pip. This is a prerequisite for using the speech-to-text functionalities.

```python
!pip install moshi
```

--------------------------------

### Install Rust Server

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Install the moshi-server crate for the Rust implementation of Kyutai STT. Use the --features cuda flag if you have a CUDA-enabled GPU.

```bash
cargo install --features cuda moshi-server
```

--------------------------------

### Run Inference with PyTorch

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Use this command to run inference on an audio file using the PyTorch implementation. Ensure the moshi package is installed.

```bash
python -m moshi.run_inference --hf-repo kyutai/stt-2.6b-en audio/bria.mp3
```

--------------------------------

### TTS with MLX (Apple Silicon) - Bash Commands

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Provides bash commands for using the moshi-mlx package for on-device TTS on Apple Silicon. It shows examples for synthesizing from stdin or a text file, including options for quantization.

```bash
# From stdin → play immediately (8-bit quantized for speed)
echo "Hey, how are you?" | python scripts/tts_mlx.py - - --quantize 8

# From text file → save WAV
python scripts/tts_mlx.py text_to_say.txt audio_output.wav

# Streaming line-by-line from stdin
echo "Hello world" | python scripts/tts_mlx_streaming.py -
```

--------------------------------

### Core MLX TTS Model Setup

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Sets up the core MLX TTS model using the moshi-mlx package. This code is intended for use on Apple Silicon devices and supports optional quantization.

```python
# Core MLX TTS model setup (from scripts/tts_mlx.py)
import mlx.core as mx, mlx.nn as nn
from moshi_mlx.models.tts import TTSModel
```

--------------------------------

### STT via Rust Server + WebSocket Client (Python)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Integrates with a Rust-based `moshi-server` for STT. The Python client streams audio via WebSocket and receives per-word timing information. Requires `moshi-server` installation and `msgpack`.

```bash
# 1. Install and start the Rust server (English+French 1B model)
cargo install --features cuda moshi-server
moshi-server worker --config configs/config-stt-en_fr-hf.toml
# Server starts at ws://127.0.0.1:8080

# 2. Transcribe from a microphone (real-time)
uv run scripts/stt_from_mic_rust_server.py

# 3. Transcribe from an audio file (simulates real-time by default)
uv run scripts/stt_from_file_rust_server.py audio/bria.mp3

# 4. Process as fast as possible (no rate limiting)
uv run scripts/stt_from_file_rust_server.py audio/bria.mp3 --rtf 1000
```

--------------------------------

### Run STT Standalone (Rust)

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Execute the standalone Rust STT implementation. Add `--timestamps` to get timing information or `--vad` to see semantic VAD output.

```bash
cd stt-rs
cargo run --features cuda -r -- ../audio/bria.mp3
```

--------------------------------

### Streaming STT with Word Timestamps (PyTorch)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

This Python script demonstrates chunk-by-chunk streaming inference for STT using `moshi.models.loaders.CheckpointInfo.from_hf_repo`, `mimi.streaming`, and `LMGen.streaming`. It decodes word boundaries to produce per-word timestamps. Ensure the necessary libraries are installed and the model is accessible.

```python
# scripts/stt_from_file_pytorch.py
import itertools, math, torch, julius, sphn
import moshi.models

info = moshi.models.loaders.CheckpointInfo.from_hf_repo("kyutai/stt-2.6b-en")
mimi      = info.get_mimi(device="cuda")
tokenizer = info.get_text_tokenizer()
lm        = info.get_moshi(device="cuda", dtype=torch.bfloat16)
lm_gen    = moshi.models.LMGen(lm, temp=0, temp_text=0.0)

audio_delay_seconds        = info.stt_config.get("audio_delay_seconds", 5.0)
audio_silence_prefix_seconds = info.stt_config.get("audio_silence_prefix_seconds", 1.0)

audio, sr = sphn.read("audio/bria.mp3")
audio = torch.from_numpy(audio).cuda()
audio = julius.resample_frac(audio, sr, mimi.sample_rate)

n_prefix = math.ceil(audio_silence_prefix_seconds * mimi.frame_rate)
n_suffix = math.ceil(audio_delay_seconds * mimi.frame_rate)
silence  = torch.zeros(1, 1, mimi.frame_size, device="cuda")

chunks = itertools.chain(
    itertools.repeat(silence, n_prefix),
    torch.split(audio[:, None], mimi.frame_size, dim=-1),
    itertools.repeat(silence, n_suffix),
)

text_tokens_accum = []
with mimi.streaming(1), lm_gen.streaming(1):
    for chunk in chunks:
        audio_tokens = mimi.encode(chunk)
        text_tokens  = lm_gen.step(audio_tokens)     # returns None until first output
        if text_tokens is not None:
            tok = text_tokens[0, 0, 0].item()
            if tok not in (0, 3):                    # 0=pad, 3=special
                print(tokenizer.id_to_piece(tok).replace("▁", " "), end="", flush=True)
            text_tokens_accum.append(text_tokens)

# Expected output (streaming to terminal):
# In the heart of an ancient forest, where the trees whispered ...

```

```bash
# Run via uv
uv run scripts/stt_from_file_pytorch.py --hf-repo kyutai/stt-2.6b-en audio/bria.mp3
```

--------------------------------

### Initialize and Generate TTS with PyTorch

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Prepare text and conditions, then generate speech using the TTSModel. Ensure model quantization is applied if needed.

```python
if quantize_bits is not None:
    nn.quantize(model.depformer, bits=quantize_bits)
    for layer in model.transformer.layers:
        nn.quantize(layer.self_attn, bits=quantize_bits)
        nn.quantize(layer.gating,    bits=quantize_bits)

tts_model = TTSModel(
    model, audio_tokenizer, text_tokenizer,
    voice_repo="kyutai/tts-voices",
    temp=0.6, cfg_coef=1,
    max_padding=8, initial_padding=2, final_padding=2,
    padding_bonus=0,
)

all_entries    = [tts_model.prepare_script([text_to_tts])]
all_attributes = [tts_model.make_condition_attributes(
    [tts_model.get_voice_path("expresso/ex03-ex01_happy_001_channel1_334s.wav")],
    cfg_coef_conditioning,
)]

result = tts_model.generate(all_entries, all_attributes, on_frame=_on_frame)
```

--------------------------------

### Download Sample Audio

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/stt_pytorch.ipynb

Download a sample audio file for testing the speech-to-text model. This MP3 file will be used as input.

```python
!wget https://github.com/kyutai-labs/moshi/raw/refs/heads/main/data/sample_fr_hibiki_crepes.mp3
```

--------------------------------

### PyTorch STT with Prompting

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Use this script to run the PyTorch STT model with text, audio, or text-audio prompts for tasks like speaker adaptation or specific formatting. The behavior is sensitive to the prompt. Requires uv.

```bash
uv run scripts/stt_from_file_pytorch_with_prompt.py \
  --hf-repo kyutai/stt-2.6b-en \
  --file bria.mp3 \
  --prompt_file ./audio/loonah.mp3 \
  --prompt_text "Loonah" \
  --cut-prompt-transcript
```

--------------------------------

### Run STT from File (Python)

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Use this script to simulate real-time processing of an audio file. Adjust the real-time factor with `--rtf` for faster processing.

```bash
uv run scripts/stt_from_file_rust_server.py audio/bria.mp3
```

--------------------------------

### Run STT Inference (MLX)

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Perform STT inference using the MLX framework on Apple silicon. Requires the `moshi-mlx` package. Set `--temp 0` for specific output.

```bash
python -m moshi_mlx.run_inference --hf-repo kyutai/stt-2.6b-en-mlx audio/bria.mp3 --temp 0
```

--------------------------------

### Run STT Model Inference on Audio File (CLI)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Use this command-line interface to run the STT model directly on a local audio file. Outputs a plain transcript to standard output. Requires `moshi>=0.2.6`.

```bash
# Install
pip install moshi

# Run inference on a local audio file
python -m moshi.run_inference --hf-repo kyutai/stt-2.6b-en audio/bria.mp3

# Using uv (no install step needed)
uvx --with moshi python -m moshi.run_inference --hf-repo kyutai/stt-2.6b-en audio/bria.mp3

# English + French bilingual model (1B, 0.5 s delay, includes semantic VAD)
python -m moshi.run_inference --hf-repo kyutai/stt-1b-en_fr audio/bria.mp3
```

--------------------------------

### Evaluate PyTorch Model on Dataset

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Run this script to evaluate the PyTorch STT model on a Hugging Face dataset and calculate performance metrics. Requires uv.

```bash
uv run scripts/evaluate_on_dataset.py  \
  --dataset meanwhile  \
  --hf-repo kyutai/stt-2.6b-en
```

--------------------------------

### Run MLX TTS with quantization

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Execute the MLX implementation for on-device inference. Use quantization flags like --quantize 8 for faster inference if the model struggles to keep up with real-time.

```bash
# From stdin, plays audio immediately
echo "Hey, how are you?" | python scripts/tts_mlx.py - - --quantize 8

# From text file to audio file
python scripts/tts_mlx.py text_to_say.txt audio_output.wav
```

--------------------------------

### Load TTS Model and Prepare Script

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/tts_pytorch.ipynb

Loads a pre-trained TTS model from a Hugging Face repository and prepares the input text script. It configures the model with specified quantization and temperature settings, and moves it to the CUDA device.

```python
# Set everything up
checkpoint_info = CheckpointInfo.from_hf_repo(DEFAULT_DSM_TTS_REPO)
tts_model = TTSModel.from_checkpoint_info(
    checkpoint_info, n_q=32, temp=0.6, device=torch.device("cuda")
)

# If you want to make a dialog, you can pass more than one turn [text_speaker_1, text_speaker_2, text_2_speaker_1, ...]
entries = tts_model.prepare_script([text], padding_between=1)
voice_path = tts_model.get_voice_path(voice)
# CFG coef goes here because the model was trained with CFG distillation,
# so it's not _actually_ doing CFG at inference time.
# Also, if you are generating a dialog, you should have two voices in the list.
condition_attributes = tts_model.make_condition_attributes([voice_path], cfg_coef=2.0)
```

--------------------------------

### Transcribe from Microphone with Rust Server

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Use this script to transcribe audio from your microphone by connecting to the running Rust STT server. Requires uv.

```bash
uv run scripts/stt_from_mic_rust_server.py
```

--------------------------------

### TTS via Rust Server + WebSocket Client (Bash)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Demonstrates how to interact with a Rust TTS server using a WebSocket client. It shows commands for synthesizing speech from stdin or a text file, and for specifying a different voice.

```bash
# Install and start TTS server (see unmute/dockerless/start_tts.sh for full setup)
moshi-server worker --config configs/config-tts.toml
# Server starts at ws://127.0.0.1:8080

# Synthesize from stdin → play immediately
echo "Hey, how are you?" | python scripts/tts_rust_server.py - -

# Synthesize from text file → save WAV
python scripts/tts_rust_server.py text_to_say.txt audio_output.wav

# Use a different voice
echo "Hello" | python scripts/tts_rust_server.py - - \
  --voice "expresso/ex03-ex01_happy_001_channel1_334s.wav"
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/stt_pytorch.ipynb

Import all required libraries for speech-to-text processing, including data handling, time, sentencepiece, sphn, textwrap, torch, and Moshi model components.

```python
from dataclasses import dataclass
import time
import sentencepiece
import sphn
import textwrap
import torch

from moshi.models import loaders, MimiModel, LMModel, LMGen
```

--------------------------------

### Load Model and Process Audio

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/stt_pytorch.ipynb

Load the speech-to-text model, tokenizer, and audio processing components. This snippet sets up the inference environment, preprocesses the audio, and runs the transcription.

```python
device = "cuda"
# Use the en+fr low latency model, an alternative is kyutai/stt-2.6b-en
checkpoint_info = loaders.CheckpointInfo.from_hf_repo("kyutai/stt-1b-en_fr")
mimi = checkpoint_info.get_mimi(device=device)
text_tokenizer = checkpoint_info.get_text_tokenizer()
lm = checkpoint_info.get_moshi(device=device)
in_pcms, _ = sphn.read("sample_fr_hibiki_crepes.mp3", sample_rate=mimi.sample_rate)
in_pcms = torch.from_numpy(in_pcms).to(device=device)

stt_config = checkpoint_info.stt_config
pad_left = int(stt_config.get("audio_silence_prefix_seconds", 0.0) * 24000)
pad_right = int((stt_config.get("audio_delay_seconds", 0.0) + 1.0) * 24000)
in_pcms = torch.nn.functional.pad(in_pcms, (pad_left, pad_right), mode="constant")
in_pcms = in_pcms[None, 0:1].expand(1, -1, -1)

state = InferenceState(mimi, text_tokenizer, lm, batch_size=1, device=device)
text = state.run(in_pcms)
print(textwrap.fill(text, width=100))
```

--------------------------------

### Streaming TTS Synthesis (PyTorch)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Manages streaming generation state for line-by-line real-time synthesis. Text lines are synthesized incrementally, and audio frames are decoded and played back frame-by-frame.

```bash
echo "Hey, how are you?" | python scripts/tts_pytorch_streaming.py -
 echo "Hey, how are you?" | python scripts/tts_pytorch_streaming.py audio_output.wav
```

--------------------------------

### Run STT from Microphone (MLX)

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Transcribe audio directly from your microphone using the MLX implementation.

```bash
python scripts/stt_from_mic_mlx.py
```

--------------------------------

### Run PyTorch TTS Streaming

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

This script provides a fully streaming implementation of PyTorch TTS. It requires the 'moshi' package.

```bash
echo "Hey, how are you?" | python scripts/tts_pytorch_streaming.py audio_output.wav
```

--------------------------------

### Extract Word Timestamps with PyTorch

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Execute this script to extract word-level timestamps from an audio file using the PyTorch implementation. Requires uv and moshi.

```bash
uv run \
  scripts/stt_from_file_pytorch.py \
  --hf-repo kyutai/stt-2.6b-en \
  audio/bria.mp3
```

--------------------------------

### Rust Server Configuration for STT

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

TOML configuration for the moshi-server binary, specifying parameters for bilingual STT models. Tune batch_size for parallel streams and asr_delay_in_tokens for latency.

```toml
authorized_ids = ["public_token"]

[modules.asr]
path = "/api/asr-streaming"
type = "BatchedAsr"
lm_model_file      = "hf://kyutai/stt-1b-en_fr-candle/model.safetensors"
text_tokenizer_file = "hf://kyutai/stt-1b-en_fr-candle/tokenizer_en_fr_audio_8000.model"
audio_tokenizer_file = "hf://kyutai/stt-1b-en_fr-candle/mimi-pytorch-e351c8d8@125.safetensors"
asr_delay_in_tokens = 6     # 0.5s delay
batch_size          = 64    # parallel streams (tuned for L40S GPU)
temperature         = 0.0

[modules.asr.model.extra_heads]
num_heads = 4
dim       = 6               # enables semantic VAD output
```

--------------------------------

### Display Audio Player

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/stt_pytorch.ipynb

Embed an audio player in the notebook to play the sample audio file. This allows for easy listening and comparison with the transcribed text.

```python
from IPython.display import Audio

Audio("sample_fr_hibiki_crepes.mp3")
```

--------------------------------

### MLX STT Inference on Apple Silicon

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Performs STT using the `moshi-mlx` package for GPU acceleration on Apple Silicon. Supports quantized models and streaming from microphone input. Includes options for VAD.

```bash
# Install
pip install moshi-mlx

# Transcribe from file (greedy decoding, temp=0)
python -m moshi_mlx.run_inference \
  --hf-repo kyutai/stt-2.6b-en-mlx audio/bria.mp3 --temp 0

# Using uv
uvx --with moshi-mlx python -m moshi_mlx.run_inference \
  --hf-repo kyutai/stt-1b-en_fr-mlx audio/bria.mp3 --temp 0

# Transcribe from microphone (streams to terminal)
python scripts/stt_from_mic_mlx.py
python scripts/stt_from_mic_mlx.py --vad   # with semantic VAD
```

--------------------------------

### Generate Audio with Streaming Callback

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/tts_pytorch.ipynb

Generates audio from the prepared text and voice conditions using a streaming approach. The `_on_frame` callback processes audio frames as they are generated, appending them to a list.

```python
print("Generating audio...")

pcms = []


def _on_frame(frame):
    print("Step", len(pcms), end="\r")
    if (frame != -1).all():
        pcm = tts_model.mimi.decode(frame[:, 1:, :]).cpu().numpy()
        pcms.append(np.clip(pcm[0, 0], -1, 1))


# You could also generate multiple audios at once by extending the following lists.
all_entries = [entries]
all_condition_attributes = [condition_attributes]
with tts_model.mimi.streaming(len(all_entries)):
    result = tts_model.generate(
        all_entries, all_condition_attributes, on_frame=_on_frame
    )

print("Done generating.")
audio = np.concatenate(pcms, axis=-1)
```

--------------------------------

### Run PyTorch TTS from stdin or file

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Use this script to generate speech from standard input or a text file. The PyTorch implementation waits for all text before generating audio.

```bash
echo "Hey, how are you?" | python scripts/tts_pytorch.py - -

# From text file to audio file
python scripts/tts_pytorch.py text_to_say.txt audio_output.wav
```

--------------------------------

### Batched STT for Dataset Evaluation (PyTorch)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Evaluates STT performance on datasets by batching audio, processing through a streaming model, and calculating WER/CER metrics. Achieves high speed on H100 GPUs. Requires `torch`, `jiwer`, and `julius`.

```python
# scripts/stt_evaluate_on_dataset.py (core loop)
import torch, jiwer, julius
import moshi.models

@torch.no_grad
def streaming_transcribe(padded_batch, mimi, lm_gen):
    bsz = padded_batch.shape[0]
    text_tokens_acc = []
    with mimi.streaming(bsz), lm_gen.streaming(bsz):
        for offset in range(0, padded_batch.shape[-1], mimi.frame_size):
            chunk = padded_batch[:, offset:offset + mimi.frame_size][:, None, :] 
            audio_tokens = mimi.encode(chunk)
            text_tokens  = lm_gen.step(audio_tokens)
            if text_tokens is not None:
                text_tokens_acc.append(text_tokens)
    return torch.concat(text_tokens_acc, axis=-1)
```

```bash
uv run scripts/stt_evaluate_on_dataset.py \
  --dataset librispeech.clean \
  --hf-repo kyutai/stt-2.6b-en \
  --batch-size 32
# Output: cer: 0.67% wer: 1.95% corpus_wer: 1.69% RTF = 68.19
```

--------------------------------

### Configure TTS Parameters

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/tts_pytorch.ipynb

Sets the input text for speech synthesis and specifies the desired voice. It also prints a link to Hugging Face for discovering available voices.

```python
# Configuration
text = "Hey there! How are you? I had the craziest day today."
voice = "expresso/ex03-ex01_happy_001_channel1_334s.wav"
print(f"See https://huggingface.co/{DEFAULT_DSM_TTS_VOICE_REPO} for available voices.")
```

--------------------------------

### Run Rust TTS Server from stdin or file

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/README.md

Interact with the running Rust TTS server to generate speech from standard input or a text file.

```bash
# From stdin, plays audio immediately
echo "Hey, how are you?" | python scripts/tts_rust_server.py - -

# From text file to audio file
python scripts/tts_rust_server.py text_to_say.txt audio_output.wav
```

--------------------------------

### Core Streaming TTS Usage (PyTorch)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Implements the core logic for streaming TTS using PyTorch. It processes text line by line, generates audio frames, and queues them for playback.

```python
# Core streaming usage (from scripts/tts_pytorch_streaming.py)
import sys, torch, sphn, numpy as np, queue
from moshi.models.tts import TTSModel, TTSGen, script_to_entries

pcms = queue.Queue()

def _on_frame(frame: torch.Tensor):
    if (frame != -1).all():
        pcm = tts_model.mimi.decode(frame[:, 1:, :]).cpu().numpy()
        pcms.put_nowait(np.clip(pcm[0, 0], -1, 1))

gen = TTSGen(tts_model, [condition_attributes], on_frame=_on_frame)

with tts_model.mimi.streaming(1):
    first_turn = True
    for line in sys.stdin:
        entries = script_to_entries(
            tts_model.tokenizer,
            tts_model.machine.token_ids,
            tts_model.mimi.frame_rate,
            [line.strip()],
            multi_speaker=first_turn and tts_model.multi_speaker,
            padding_between=1,
        )
        first_turn = False
        for entry in entries:
            gen.append_entry(entry)
            gen.process()        # generates until buffer allows
    gen.process_last()           # flush remaining frames
```

--------------------------------

### Display Generated Audio

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/tts_pytorch.ipynb

Displays the generated audio using IPython's Audio widget, setting the correct sample rate from the TTS model's configuration and enabling autoplay.

```python
display(Audio(audio, rate=tts_model.mimi.sample_rate, autoplay=True))
```

--------------------------------

### Rust Server Configuration for TTS

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

TOML configuration for the TTS Rust server. Adjust n_q for audio quality, cfg_coef for voice adherence, and padding_bonus for speech rate.

```toml
authorized_ids = ["public_token"]

[modules.tts_py]
type       = "Py"
path       = "/api/tts_streaming"
batch_size = 8               # parallel TTS streams
text_bos_token = 1

[modules.tts_py.py]
voice_folder  = "hf-snapshot://kyutai/tts-voices/**/*.safetensors"
default_voice = "unmute-prod-website/default_voice.wav"
cfg_coef      = 2.0          # recommended for tts-1.6b-en_fr (CFG distillation)
cfg_is_no_text = true
padding_between = 1          # pause between words; improves articulation
n_q             = 24         # RVQ levels: higher=better quality, slower (max 32)
padding_bonus   = 0          # speech rate: positive=slower, negative=faster
```

--------------------------------

### Rust STT Binary Compilation and Execution

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Compiles and runs a standalone Rust STT binary. Supports CUDA, Metal, and CPU backends. Can output plain text, word-level timestamps, or VAD probabilities. Allows specifying different Hugging Face repositories.

```bash
cd stt-rs
# Plain transcript
cargo run --features cuda -r -- ../audio/bria.mp3

# With word-level timestamps
cargo run --features cuda -r -- ../audio/bria.mp3 --timestamps
# Output: [ 0.02- 0.48] In
#         [ 0.48- 0.64] the
#         ...

# With semantic VAD output
cargo run --features cuda -r -- ../audio/bria.mp3 --vad
# Output: In the heart ... <endofturn pr=0.87>

# Use a different HF repo
cargo run --features cuda -r -- ../audio/bria.mp3 \
  --hf-repo kyutai/stt-2.6b-en-candle
```

--------------------------------

### Import Libraries for TTS

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/tts_pytorch.ipynb

Imports essential libraries including NumPy, PyTorch, and specific modules from the Moshi library for TTS model loading and usage. Also imports IPython display utilities.

```python
import numpy as np
import torch
from moshi.models.loaders import CheckpointInfo
from moshi.models.tts import DEFAULT_DSM_TTS_REPO, DEFAULT_DSM_TTS_VOICE_REPO, TTSModel

from IPython.display import display, Audio
```

--------------------------------

### Configure Sentencepiece Compilation with GCC

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/FAQ.md

When compiling sentencepiece rust bindings, if you encounter errors with recent gcc versions (e.g., gcc 15), use gcc-13 by setting these environment variables. This also includes setting CC and CXX for cargo.

```bash
export CMAKE_C_COMPILER=/usr/bin/gcc-13
export CMAKE_CXX_COMPILER=/usr/bin/g++-13 
CC=gcc-13 CXX=g++-13 cargo build --release
```

--------------------------------

### Streaming STT with Semantic VAD (PyTorch)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Enables the semantic VAD head to detect the end of a user's turn. Requires the `kyutai/stt-1b-en_fr` model and `moshi` library.

```python
lm_gen = moshi.models.LMGen(lm, temp=0, temp_text=0.0)

with mimi.streaming(1), lm_gen.streaming(1):
    for chunk in chunks:
        audio_tokens = mimi.encode(chunk)
        text_tokens, vad_heads = lm_gen.step_with_extra_heads(audio_tokens)

        if vad_heads:
            pr_vad = vad_heads[2][0, 0, 0].cpu().item()
            if pr_vad > 0.5:
                print(" [end of turn detected]")

        tok = text_tokens[0, 0, 0].cpu().item()
        if tok not in (0, 3):
            print(tokenizer.id_to_piece(tok).replace("▁", " "), end="", flush=True)
```

--------------------------------

### Generate Audio from Pre-computed Voice Embedding

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Loads a pre-computed voice embedding and generates audio frames. The generated frames are then decoded using Mimi and saved to a WAV file.

```python
voice_path = tts_model.get_voice_path(
    "expresso/ex03-ex01_happy_001_channel1_334s.wav")
condition_attributes = tts_model.make_condition_attributes(
    [voice_path], cfg_coef=2.0)   # cfg_coef=2.0 is recommended for tts-1.6b-en_fr

# Generate — returns all audio frames
result = tts_model.generate([entries], [condition_attributes])

# Decode audio frames with Mimi
with tts_model.mimi.streaming(1), torch.no_grad():
    pcms = []
    for frame in result.frames[tts_model.delay_steps:]:
        pcm = tts_model.mimi.decode(frame[:, 1:, :]).cpu().numpy()
        pcms.append(np.clip(pcm[0, 0], -1, 1))

sphn.write_wav("output.wav", np.concatenate(pcms, axis=-1),
               tts_model.mimi.sample_rate)
# Produces: output.wav at 24 kHz
```

--------------------------------

### PyTorch TTS Generation

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Generates speech from text using a PyTorch TTS model. Supports input from stdin or text files, and output to WAV files or direct playback. Requires `moshi` and `sphn` libraries.

```bash
# From stdin → play immediately
echo "Hey, how are you?" | python scripts/tts_pytorch.py - -

# From text file → save to WAV
python scripts/tts_pytorch.py text_to_say.txt audio_output.wav

# Using uv (no install)
echo "Hello world" | uvx --with moshi python scripts/tts_pytorch.py - -
```

--------------------------------

### Prompted STT with PromptHook (PyTorch)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Steers the model's output for spelling, speaker, or language by enforcing a text prefix. The `PromptHook` masks logits to ensure the prefix is consumed correctly. Requires `torch` and `collections.deque`.

```python
# scripts/stt_from_file_with_prompt_pytorch.py
from collections import deque
import torch

class PromptHook:
    def __init__(self, tokenizer, prefix, padding_tokens=(0, 3)):
        self.tokenizer = tokenizer
        self.prefix_enforce = deque(tokenizer.encode(prefix))
        self.padding_tokens = padding_tokens

    def on_logits(self, logits):
        if not self.prefix_enforce:
            return
        mask = torch.zeros_like(logits, dtype=torch.bool)
        for t in self.padding_tokens:
            mask[..., t] = True
        mask[..., self.prefix_enforce[0]] = True
        logits[:] = torch.where(mask, logits, float("-inf"))

    def on_token(self, token):
        if not self.prefix_enforce:
            return
        if token.item() not in self.padding_tokens:
            self.prefix_enforce.popleft()

prompt_hook = PromptHook(tokenizer, "Loonah")
lm_gen = moshi.models.LMGen(
    lm, temp=0, temp_text=0.0,
    on_text_hook=prompt_hook.on_token,
    on_text_logits_hook=prompt_hook.on_logits,
)
```

```bash
# CLI usage — steer the model to use "Loonah" spelling instead of "Luna"
uv run scripts/stt_from_file_with_prompt_pytorch.py \
  --hf-repo kyutai/stt-2.6b-en \
  --file audio/bria.mp3 \
  --prompt_file audio/loona.mp3 \
  --prompt_text "Loonah" \
  --cut-prompt-transcript
# Output: In the heart of an ancient forest ... a peculiar rabbit named Loonah ...
```

--------------------------------

### MLX Streaming STT Core Loop

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

The core MLX streaming loop for STT, handling audio input from `sounddevice`, tokenization, and generation with `moshi-mlx`. Prints transcribed text to the console.

```python
# Core MLX streaming loop (scripts/stt_from_mic_mlx.py)
import rustymimi, sounddevice as sd, sentencepiece, queue
import mlx.core as mx
from moshi_mlx import models, utils

gen = models.LmGen(
    model=model,
    max_steps=4096,
    text_sampler=utils.Sampler(top_k=25, temp=0),
    audio_sampler=utils.Sampler(top_k=250, temp=0.8),
    check=False,
)

block_queue = queue.Queue()
with sd.InputStream(channels=1, dtype="float32",
                    samplerate=24000, blocksize=1920,
                    callback=lambda d, *_: block_queue.put(d.copy())):
    while True:
        block = block_queue.get()[None, :, 0]
        audio_tokens = audio_tokenizer.encode_step(block[None, 0:1])
        audio_tokens = mx.array(audio_tokens).transpose(0, 2, 1)[:, :, :other_codebooks]
        text_token = gen.step(audio_tokens[0]).item()
        if text_token not in (0, 3):
            print(text_tokenizer.id_to_piece(text_token).replace("▁", " "),
                  end="", flush=True)
```

--------------------------------

### PyTorch TTS Core Usage

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Core PyTorch TTS usage for preparing script tokens and generating audio frames. Loads a TTS model from Hugging Face, prepares input text, and sets up generation parameters.

```python
# Core PyTorch TTS usage (from scripts/tts_pytorch.py)
import torch, sphn, numpy as np
from moshi.models.loaders import CheckpointInfo
from moshi.models.tts import TTSModel, DEFAULT_DSM_TTS_REPO, DEFAULT_DSM_TTS_VOICE_REPO

checkpoint_info = CheckpointInfo.from_hf_repo(DEFAULT_DSM_TTS_REPO)
tts_model = TTSModel.from_checkpoint_info(
    checkpoint_info, n_q=32, temp=0.6, device="cuda"
)

text = "In the heart of an ancient forest, something magical happened."

# Prepare script tokens (supports multi-turn dialogs as a list)
entries = tts_model.prepare_script([text], padding_between=1)

```

--------------------------------

### WebSocket TTS Streaming Protocol (Python)

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Implements a Python WebSocket client to communicate with a Rust TTS server. It sends text word-by-word and receives audio chunks, then saves them to a WAV file.

```python
# WebSocket TTS streaming protocol (from scripts/tts_rust_server.py)
import asyncio, msgpack, websockets
from urllib.parse import urlencode

async def tts_client(url, api_key, text, output_wav):
    params = {"voice": "expresso/ex03-ex01_happy_001_channel1_334s.wav",
              "format": "PcmMessagePack"}
    uri = f"{url}/api/tts_streaming?{urlencode(params)}"
    # Auth via header OR query string: append &auth_id={api_key} to uri
    async with websockets.connect(uri,
            additional_headers={"kyutai-api-key": api_key}) as ws:

        # Send text word-by-word
        for word in text.split():
            await ws.send(msgpack.packb({"type": "Text", "text": word}))
        await ws.send(msgpack.packb({"type": "Eos"}))

        frames = []
        async for msg_bytes in ws:
            msg = msgpack.unpackb(msg_bytes)
            if msg["type"] == "Audio":
                frames.append(np.array(msg["pcm"], dtype=np.float32))

        sphn.write_wav(output_wav, np.concat(frames, -1), 24000)

asyncio.run(tts_client(
    "ws://127.0.0.1:8080", "public_token",
    "Hello, this is a test.", "output.wav"))
```

--------------------------------

### Define InferenceState Class

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/stt_pytorch.ipynb

The `InferenceState` class encapsulates the speech-to-text inference pipeline. It initializes the MimiModel, SentencePiece tokenizer, and LMGen for text generation, and provides a `run` method to process audio input and return transcribed text.

```python
@dataclass
class InferenceState:
    mimi: MimiModel
    text_tokenizer: sentencepiece.SentencePieceProcessor
    lm_gen: LMGen

    def __init__(
        self,
        mimi: MimiModel,
        text_tokenizer: sentencepiece.SentencePieceProcessor,
        lm: LMModel,
        batch_size: int,
        device: str | torch.device,
    ):
        self.mimi = mimi
        self.text_tokenizer = text_tokenizer
        self.lm_gen = LMGen(lm, temp=0, temp_text=0, use_sampling=False)
        self.device = device
        self.frame_size = int(self.mimi.sample_rate / self.mimi.frame_rate)
        self.batch_size = batch_size
        self.mimi.streaming_forever(batch_size)
        self.lm_gen.streaming_forever(batch_size)

    def run(self, in_pcms: torch.Tensor):
        ntokens = 0
        first_frame = True
        chunks = [
            c
            for c in in_pcms.split(self.frame_size, dim=2)
            if c.shape[-1] == self.frame_size
        ]
        start_time = time.time()
        all_text = []
        for chunk in chunks:
            codes = self.mimi.encode(chunk)
            if first_frame:
                # Ensure that the first slice of codes is properly seen by the transformer
                # as otherwise the first slice is replaced by the initial tokens.
                tokens = self.lm_gen.step(codes)
                first_frame = False
            tokens = self.lm_gen.step(codes)
            if tokens is None:
                continue
            assert tokens.shape[1] == 1
            one_text = tokens[0, 0].cpu()
            if one_text.item() not in [0, 3]:
                text = self.text_tokenizer.id_to_piece(one_text.item())
                text = text.replace("▁", " ")
                all_text.append(text)
            ntokens += 1
        dt = time.time() - start_time
        print(
            f"processed {ntokens} steps in {dt:.0f}s, {1000 * dt / ntokens:.2f}ms/step"
        )
        return "".join(all_text)
```

--------------------------------

### Set CXXFLAGS for Sentencepiece

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/FAQ.md

Alternatively, to resolve compilation errors with sentencepiece rust bindings due to recent gcc versions, set the CXXFLAGS environment variable.

```bash
export CXXFLAGS="-include cstdint"
```

--------------------------------

### Disable Torch Compilation

Source: https://github.com/kyutai-labs/delayed-streams-modeling/blob/main/FAQ.md

Set this environment variable to disable torch compilation if you encounter compilation errors with certain PyTorch/triton versions.

```bash
export NO_TORCH_COMPILE=1
```

--------------------------------

### Python WebSocket STT Stream

Source: https://context7.com/kyutai-labs/delayed-streams-modeling/llms.txt

Streams audio data over WebSockets for real-time transcription. Requires a WebSocket server and msgpack for serialization. Sends audio chunks and receives transcribed text or markers.

```python
import asyncio, msgpack, websockets

async def stream_audio(url, api_key, audio_data):
    headers = {"kyutai-api-key": api_key}
    # Alternatively: append ?auth_id={api_key} to the URL
    async with websockets.connect(url, additional_headers=headers) as ws:
        # Send 1 second of silence prefix (required for 2.6B model)
        await ws.send(msgpack.packb(
            {"type": "Audio", "pcm": [0.0] * 24000}, use_single_float=True))

        for i in range(0, len(audio_data), 1920):
            await ws.send(msgpack.packb(
                {"type": "Audio", "pcm": audio_data[i:i+1920].tolist()},
                use_single_float=True))

        # Send end-of-stream marker; server echoes it back after flushing
        await ws.send(msgpack.packb({"type": "Marker", "id": 0}, use_single_float=True))

        async for message in ws:
            data = msgpack.unpackb(message, raw=False)
            if data["type"] == "Word":
                print(data["text"], end=" ", flush=True)
            elif data["type"] == "Marker":
                break  # all audio processed

transcript = asyncio.run(
    stream_audio("ws://127.0.0.1:8080/api/asr-streaming", "public_token", audio_array))
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.