### Install Moshi PyTorch Client

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Install the Moshi PyTorch client from PyPI. This is the standard way to get the PyTorch backend.

```bash
pip install -U moshi
```

--------------------------------

### Web UI Client Setup and Development

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Instructions for installing dependencies, running the development server with hot-reloading, and building the production version of the React/TypeScript web client.

```bash
# Install dependencies
cd client
nvm use          # uses .nvmrc to select correct Node version
npm install

# Development server
npm run dev      # hot-reload at https://localhost:5173

# Production build (output in client/dist/)
npm run build

# Skip queue for standalone mode
# Navigate to: https://localhost:5173/?worker_addr=0.0.0.0:8998
```

--------------------------------

### Launch Gradio Demo

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Starts a Gradio demo for Moshi. Requires gradio-webrtc to be installed. Replace <moshi-server-url> with the actual server address.

```bash
python -m moshi.client_gradio --url <moshi-server-url>
```

--------------------------------

### Install Moshi and Gradio

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/demo_moshi.ipynb

Install the Moshi library from its GitHub repository and the Gradio library for web UI interactions.

```python
!pip install "git+https://git@github.com/kyutai-labs/moshi.git#egg=moshi&subdirectory=moshi"
!pip install gradio
```

--------------------------------

### Install Moshi with Development Dependencies

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

This command installs Moshi from a local clone with development dependencies, suitable for contributing to the project. It also installs pre-commit hooks for code quality.

```bash
pip install -e '.[dev]'
pre-commit install
```

--------------------------------

### Install Moshi MLX Backend

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Install the Moshi MLX backend for on-device inference on macOS/Apple Silicon. Python 3.12 is recommended. Supports stable or bleeding-edge installation.

```bash
pip install -U moshi_mlx      # requires Python 3.12 recommended
# or bleeding edge:
pip install -U -e "git+https://git@github.com/kyutai-labs/moshi.git#egg=moshi_mlx&subdirectory=moshi_mlx"
```

--------------------------------

### Install Moshi for Development

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Install Moshi and its development dependencies from a local clone of the repository. Ensure you are in the root directory of the cloned repository.

```bash
pip install -e 'moshi[dev]'
pip install -e 'moshi_mlx[dev]'
pre-commit install
```

--------------------------------

### Install Moshi MLX Client

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Install the Moshi MLX client from PyPI. This client is best used with Python 3.12.

```bash
pip install -U moshi_mlx
```

--------------------------------

### Install Moshi PyTorch

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

Install Moshi PyTorch from PyPI or from the bleeding-edge Git repository. Ensure PyTorch is installed with the correct CUDA version if needed.

```bash
pip install -U moshi      # moshi PyTorch, from PyPI
# Or the bleeding edge versions for Moshi
pip install -U -e "git+https://git@github.com/kyutai-labs/moshi#egg=moshi&subdirectory=moshi"
```

--------------------------------

### Launch Moshi Gradio Demo Client

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Launch a browser-accessible Gradio demo client that connects to any specified Moshi server URL. Requires installation of gradio-webrtc.

```bash
pip install "gradio-webrtc>=0.0.18"
python -m moshi.client_gradio --url http://localhost:8998
```

--------------------------------

### Start Moshi Client

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

Connect to a running Moshi server using the command-line client. Note that this client is barebone and lacks features like echo cancellation.

```bash
python -m moshi.client [--url URL_TO_GRADIO]
```

--------------------------------

### Install Moshi PyTorch Backend

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Install the stable release or the bleeding-edge version of the Moshi PyTorch backend using pip.

```bash
pip install -U moshi          # stable release
# or bleeding edge:
pip install -U -e "git+https://git@github.com/kyutai-labs/moshi.git#egg=moshi&subdirectory=moshi"
```

--------------------------------

### Start PyTorch Server

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Launches the PyTorch-based Moshi server. Use --gradio-tunnel for remote access or --hf-repo to specify a different model.

```bash
python -m moshi.server [--gradio-tunnel] [--hf-repo kyutai/moshika-pytorch-bf16]
```

--------------------------------

### Start Moshi Server

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

Start the Moshi server to run the model. Access the web UI at localhost:8998. Use --gradio-tunnel for remote access or --hf-repo to specify a different pretrained model.

```bash
python -m moshi.server [--gradio-tunnel]
```

--------------------------------

### Install rustymimi Python Package

Source: https://github.com/kyutai-labs/moshi/blob/main/rust/README.md

Install the rustymimi package using pip. Alternatively, compile from source using maturin.

```bash
pip install rustymimi
```

```bash
maturin dev -r -m rust/mimi-pyo3/Cargo.toml
```

--------------------------------

### Build Web UI Client

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Builds the web UI client for Moshi. Requires npm to be installed. Navigate to the client directory first.

```bash
cd client
npm install
npm run build
```

--------------------------------

### Install Rust Mimi Implementation

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Install the Rust implementation of Mimi with Python bindings from PyPI. This is a dependency for moshi_mlx.

```bash
pip install rustymimi
```

--------------------------------

### Browser WebSocket Client Example

Source: https://context7.com/kyutai-labs/moshi/llms.txt

JavaScript example demonstrating how to connect to a Moshi WebSocket server, send a handshake, and handle audio, text, and error messages.

```javascript
// Browser WebSocket client example
const ws = new WebSocket("wss://localhost:8998/api/chat");

ws.onopen = () => {
  // Send handshake (MT=0)
  const buf = new ArrayBuffer(9);
  const view = new DataView(buf);
  view.setUint8(0, 0);          // MT=0 Handshake
  view.setUint32(1, 0, true);   // protocol_version = 0
  view.setUint32(5, 0, true);   // model_version = 0
  ws.send(buf);
};

ws.onmessage = async (event) => {
  const buf = await event.data.arrayBuffer();
  const view = new DataView(buf);
  const mt = view.getUint8(0);

  if (mt === 1) {
    // Audio frame — decode ogg/opus and play
    const audioData = buf.slice(1);
    // ... feed to Web Audio API decoder
  } else if (mt === 2) {
    // Text token from Moshi's inner monologue
    const text = new TextDecoder().decode(buf.slice(1));
    console.log("Moshi says:", text);
  } else if (mt === 5) {
    console.error("Server error:", new TextDecoder().decode(buf.slice(1)));
  }
};

// Send user audio (MT=1)
function sendAudioFrame(oggOpusBytes) {
  const mt = new Uint8Array([1]);
  const payload = new Uint8Array(oggOpusBytes);
  const msg = new Uint8Array(1 + payload.length);
  msg.set(mt);
  msg.set(payload, 1);
  ws.send(msg.buffer);
}
```

--------------------------------

### Install Bleeding Edge Moshi PyTorch

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Install the latest development version of the Moshi PyTorch client directly from the GitHub repository.

```bash
pip install -U -e "git+https://git@github.com/kyutai-labs/moshi.git#egg=moshi&subdirectory=moshi"
```

--------------------------------

### Install Moshi MLX

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi_mlx/README.md

Install the Moshi MLX package from PyPI or from the bleeding-edge Git repository. Ensure you have Python 3.12 for the best experience.

```bash
pip install moshi_mlx  # moshi MLX, from PyPI, best with Python 3.12.
# Or the bleeding edge versions for Moshi and Moshi-MLX.
pip install -e "git+https://git@github.com/kyutai-labs/moshi#egg=moshi_mlx&subdirectory=moshi_mlx"
```

--------------------------------

### Start Moshi PyTorch Server

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Launch the WebSocket inference server for Moshi. By default, it loads the 'Moshiko' model. Options are available to specify different models, use Gradio tunnels for remote access, or reuse tunnel tokens.

```bash
# Default model (Moshiko, male voice, bf16)
python -m moshi.server

# Female voice model
python -m moshi.server --hf-repo kyutai/moshika-pytorch-bf16

# Remote GPU with no direct browser access — create a Gradio tunnel
python -m moshi.server --gradio-tunnel

# Reuse the same tunnel URL across restarts
python -m moshi.server --gradio-tunnel --gradio-tunnel-token MY_SECRET_TOKEN

# Access locally (after SSH port-forward or on the same machine)
# Navigate to http://localhost:8998
```

--------------------------------

### Install Bleeding Edge Moshi MLX

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Install the latest development version of the Moshi MLX client directly from the GitHub repository.

```bash
pip install -U -e "git+https://git@github.com/kyutai-labs/moshi.git#egg=moshi_mlx&subdirectory=moshi_mlx"
```

--------------------------------

### Install Rust-backed Mimi Python Bindings

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Install the pre-built wheel for the Rust-backed Mimi Python bindings (Python 3.12). Alternatively, compile locally using Maturin if you have the Rust toolchain.

```bash
pip install rustymimi          # pre-built wheel (Python 3.12)
# or compile locally (requires Rust toolchain):
pip install maturin
maturin dev -r -m rust/mimi-pyo3/Cargo.toml
```

--------------------------------

### Build rustymimi Locally

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Build the `rustymimi` component locally using `maturin`. This requires Rust to be properly installed. Ensure you have `maturin` installed first.

```bash
pip install maturin
maturin dev -r -m rust/mimi-pyo3/Cargo.toml
```

--------------------------------

### Start PyTorch Inference Server

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Starts a WebSocket inference server that streams bidirectional audio in real time. The server loads a pretrained model and listens on port 8998. The web UI is served from the same endpoint.

```APIDOC
## Start PyTorch Inference Server

### Description
Starts a WebSocket inference server that streams bidirectional audio in real time. The server loads a pretrained model and listens on port 8998. The web UI is served from the same endpoint.

### Command
```bash
python -m moshi.server
```

### Options
- `--hf-repo <repository>`: Specify a Hugging Face repository for the model (e.g., `kyutai/moshika-pytorch-bf16` for female voice).
- `--gradio-tunnel`: Create a Gradio tunnel for remote access.
- `--gradio-tunnel-token <token>`: Use a specific token for the Gradio tunnel.

### Example Usage
```bash
# Default model (Moshiko, male voice, bf16)
python -m moshi.server

# Female voice model
python -m moshi.server --hf-repo kyutai/moshika-pytorch-bf16

# Remote GPU with no direct browser access — create a Gradio tunnel
python -m moshi.server --gradio-tunnel

# Reuse the same tunnel URL across restarts
python -m moshi.server --gradio-tunnel --gradio-tunnel-token MY_SECRET_TOKEN
```
```

--------------------------------

### Docker Compose for CUDA

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Starts the Moshi services using Docker Compose, specifically for CUDA environments. Requires NVIDIA Container Toolkit.

```bash
docker compose up
```

--------------------------------

### Stream Audio Frame-by-Frame with Mimi Encode

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Encode audio frame-by-frame using Mimi in streaming mode. Ensure all frames are the same size and a multiple of `frame_size`. This example demonstrates basic encoding and encoding with an execution mask for desynchronized batches.

```python
import torch
from moshi.models import loaders

mimi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MIMI_NAME)
mimi = loaders.get_mimi(mimi_weight, device='cpu')
mimi.set_num_codebooks(8)

wav = torch.randn(1, 1, 24000 * 5)  # 5 seconds
frame_size = mimi.frame_size         # 1920 samples at 24kHz → 80ms

all_codes = []
with torch.no_grad(), mimi.streaming(batch_size=1):
    for offset in range(0, wav.shape[-1], frame_size):
        frame = wav[:, :, offset: offset + frame_size]
        codes = mimi.encode(frame)   # → [1, 8, 1] per frame
        assert codes.shape[-1] == 1
        all_codes.append(codes)

print(f"Total frames encoded: {len(all_codes)}")  # 62 frames for 5s

# Streaming with execution mask (desynchronized batches)
with torch.no_grad(), mimi.streaming(batch_size=4):
    mask = torch.tensor([False, True, False, True])
    mimi.set_exec_mask(mask)
    frame = torch.randn(4, 1, frame_size)
    codes = mimi.encode(frame)
    # Entries 0 and 2 are masked — their internal state is unchanged
```

--------------------------------

### Streaming Execution Mask for Desynchronized Batches

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

This example shows how to use an execution mask with Mimi's streaming functionality to handle desynchronized batches. It indicates which inputs are valid for processing, ensuring internal state is only updated for valid entries.

```python
with torch.no_grad(), mimi.streaming(4):
    mask = torch.tensor([False, True, False, True])
    mimi.set_exec_mask(mask)
    frame = torch.randn(4, 1, mimi.frame_size)
    codes = mimi.encode(frame)
    # From the point of view of the first and third entries, nothing has happen.
    # The codes for those two should simply be discarded.
```

--------------------------------

### Gradio Demo Client

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Launches a browser-accessible Gradio demo that connects to any Moshi server URL.

```APIDOC
## Gradio Demo Client

### Description
Launches a browser-accessible Gradio demo that connects to any Moshi server URL.

### Installation
```bash
pip install "gradio-webrtc>=0.0.18"
```

### Command
```bash
python -m moshi.client_gradio --url <server_url>
```

### Parameters
- `--url <server_url>`: The URL of the Moshi server to connect to (e.g., `http://localhost:8998`).

### Example Usage
```bash
python -m moshi.client_gradio --url http://localhost:8998
```
```

--------------------------------

### Run Moshi WebUI Server

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/demo_moshi.ipynb

Launch the Moshi WebUI using Gradio for live translation. This command uses a quantized PyTorch model and enables Gradio tunneling.

```bash
! python -m moshi.server --gradio-tunnel --hf-repo kyutai/moshiko-pytorch-q8 --half
```

--------------------------------

### Compile and Run Rust Backend (CUDA/Metal)

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Build and execute the production-grade Rust inference server. Supports CUDA on Linux/Windows and Metal on macOS. Requires generating TLS certificates first.

```bash
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
    -days 365 -nodes -subj "/CN=localhost"
```

```bash
cd rust
cargo run --features cuda --bin moshi-backend -r -- \
    --config moshi-backend/config.json standalone
```

```bash
cargo run --features cuda --bin moshi-backend -r -- \
    --config moshi-backend/config.json standalone
```

```bash
cargo run --features metal --bin moshi-backend -r -- \
    --config moshi-backend/config.json standalone
```

```bash
# Visit https://localhost:8998 (accept the self-signed cert warning)
```

--------------------------------

### Run MLX Local Web UI (macOS)

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Launch Moshi with a browser-based web UI served locally. Ensure Chrome or Safari is used for microphone access.

```bash
python -m moshi_mlx.local_web
# Opens http://localhost:8998 — use Chrome/Safari for microphone access
```

--------------------------------

### Run Local Moshi MLX Models

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi_mlx/README.md

Execute the local Moshi MLX interface from the command line. Use the `-q` flag to specify quantization bits and `--hf-repo` to select a Hugging Face model repository. Ensure the quantization level matches the model's repository.

```bash
python -m moshi_mlx.local -q 4   # weights quantized to 4 bits
python -m moshi_mlx.local -q 8   # weights quantized to 8 bits
# And using a different pretrained model:
python -m moshi_mlx.local -q 4 --hf-repo kyutai/moshika-mlx-q4
python -m moshi_mlx.local -q 8 --hf-repo kyutai/moshika-mlx-q8
# be careful to always match the `-q` and `--hf-repo` flag.
```

--------------------------------

### Run Moshi CLI Client (Terminal UI)

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Connect to any running Moshi server using the text-based terminal interface. Navigate to the 'rust' directory before running.

```bash
cd rust
cargo run --bin moshi-cli -r -- tui --host localhost
```

--------------------------------

### Run Moshi WebUI Server with Moshika's Voice

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/demo_moshi.ipynb

Launch the Moshi WebUI with Moshika's voice using a specific PyTorch model and BF16 precision. The command filters out 'frame handled' logs.

```bash
# Or with Moshika's voice
# ! python -m moshi.server --gradio-tunnel --hf-repo kyutai/moshika-1b-pytorch-bf16 --half | grep -v 'frame handled'
```

--------------------------------

### Connect Moshi CLI Client

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Use the barebones command-line client to connect to a running Moshi server. This client does not include echo cancellation or lag compensation.

```bash
# Connect to local server
python -m moshi.client

# Connect to a remote Gradio tunnel URL
python -m moshi.client --url https://<tunnel-id>.gradio.live
```

--------------------------------

### Run MLX Backend Locally (macOS)

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Execute Moshi on-device using MLX. Supports 4-bit and 8-bit quantization for memory efficiency. Specify different models using the `--hf-repo` flag.

```bash
python -m moshi_mlx.local -q 4
```

```bash
python -m moshi_mlx.local -q 8
```

```bash
python -m moshi_mlx.local -q 4 --hf-repo kyutai/moshika-mlx-q4
```

```bash
python -m moshi_mlx.local -q 8 --hf-repo kyutai/moshika-mlx-q8
```

```bash
python -m moshi_mlx.local -q 0 --hf-repo kyutai/moshika-mlx-bf16
```

--------------------------------

### Moshi Language Model Generation

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

Shows how to load the Moshi language model and use it for generating audio, typically after encoding audio with Mimi.

```APIDOC
## Moshi Language Model Generation

### Description
This section details how to load the Moshi language model (LM) and utilize the `LMGen` class for generating audio based on encoded inputs. This is typically used after obtaining codes from the Mimi model.

### Usage

1. **Load Moshi LM**: Download pre-trained weights and load the Moshi language model.
2. **Initialize `LMGen`**: Create an instance of `LMGen` with the loaded Moshi model and desired sampling parameters (temperature).
3. **Generate Audio**: Use the `LMGen` instance to generate audio chunks.

### Code Example
```python
from huggingface_hub import hf_hub_download
import torch

from moshi.models import loaders, LMGen

# Assuming Mimi is already loaded and used to get 'codes'
# mimi = ...
# codes = mimi.encode(wav)

# Load Moshi LM and move to GPU
mimi.cuda() # Example: move Mimi to GPU if available
moshi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MOSHI_NAME)
moshi = loaders.get_moshi_lm(moshi_weight, device='cuda')

# Initialize LMGen for generation
lm_gen = LMGen(moshi, temp=0.8, temp_text=0.7)  # Handles sampling parameters

out_wav_chunks = []
# Example of generation (actual generation logic would follow)
# with torch.no_grad():
#     for chunk in lm_gen.generate_stream(codes):
#         out_wav_chunks.append(chunk)
```

### Parameters

- **`moshi_weight`** (str): Path to the downloaded Moshi language model weights.
- **`device`** (str): Device to load the model on ('cpu' or 'cuda').
- **`moshi`** (torch.nn.Module): The loaded Moshi language model.
- **`temp`** (float): Temperature parameter for sampling during generation.
- **`temp_text`** (float): Temperature parameter for text-based sampling (if applicable).

### Response

- **`out_wav_chunks`** (list): A list containing generated audio chunks (torch.Tensor).

```

--------------------------------

### Run Moshi Rust Client CLI

Source: https://github.com/kyutai-labs/moshi/blob/main/rust/README.md

Launch the Moshi command-line interface in TUI mode from the rust directory.

```bash
cargo run --bin moshi-cli -r -- tui --host localhost
```

--------------------------------

### CLI Client

Source: https://context7.com/kyutai-labs/moshi/llms.txt

A barebones command-line audio client that connects to a running Moshi server. It does not include echo cancellation or lag compensation.

```APIDOC
## CLI Client

### Description
A barebones command-line audio client that connects to a running Moshi server. It does not include echo cancellation or lag compensation.

### Command
```bash
python -m moshi.client
```

### Options
- `--url <url>`: The URL of the Moshi server to connect to (e.g., `https://<tunnel-id>.gradio.live`).

### Example Usage
```bash
# Connect to local server
python -m moshi.client

# Connect to a remote Gradio tunnel URL
python -m moshi.client --url https://<tunnel-id>.gradio.live
```
```

--------------------------------

### Load and Use Mimi Model (Streaming)

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

Demonstrates streaming audio encoding with the Mimi model. Ensure audio is a multiple of frame size (1920) for streaming. GPU usage requires consistent audio frame sizes due to CUDAGraphs.

```python
# Supports streaming too.
frame_size = mimi.frame_size
all_codes = []
with mimi.streaming(batch_size=1):
    for offset in range(0, wav.shape[-1], frame_size):
        frame = wav[:, :, offset: offset + frame_size]
        codes = mimi.encode(frame)
        assert codes.shape[-1] == 1, codes.shape
        all_codes.append(codes)

## WARNING: When streaming, make sure to always feed a total amount of audio that is a multiple
#           of the frame size (1920). You should pad or buffer accordingly. Since version 0.2.5a,
        Mimi no longer supports partial frames in streaming mode. Besides, when executing on GPU,
        you should always pass the same amount of audio, as the calls are CUDAGraphed for efficiency.
```

--------------------------------

### Load Model Artifacts from Hugging Face Repository

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Use `CheckpointInfo.from_hf_repo` to download all necessary model artifacts, including LM weights, Mimi weights, and tokenizer, from a specified Hugging Face repository. This factory method simplifies model loading and allows for overriding specific weight files.

```python
from moshi.models import loaders
import torch

# Load Moshika (female voice)
checkpoint = loaders.CheckpointInfo.from_hf_repo("kyutai/moshika-pytorch-bf16")

mimi = checkpoint.get_mimi(device='cuda')
lm   = checkpoint.get_moshi(device='cuda', dtype=torch.bfloat16)
tokenizer = checkpoint.get_text_tokenizer()  # SentencePieceProcessor

print(checkpoint.model_type)   # "moshi"
print(checkpoint.lm_gen_config)

# Load quantized int8 model (Rust/Candle only)
checkpoint_q8 = loaders.CheckpointInfo.from_hf_repo("kyutai/moshiko-candle-q8")

# Override specific weight files (e.g., point to a local fine-tune)
checkpoint_custom = loaders.CheckpointInfo.from_hf_repo(
    "kyutai/moshika-pytorch-bf16",
    moshi_weights="file:///path/to/my/finetuned_model.safetensors",
)
```

--------------------------------

### Mimi Audio Encoding and Decoding

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

Demonstrates how to load the Mimi model, encode audio into codes, and decode codes back into audio. Supports both batch and streaming modes.

```APIDOC
## Mimi Audio Encoding and Decoding

### Description
This section shows how to load the Mimi model, encode audio waveforms into discrete codes, and decode these codes back into audio waveforms. It covers both standard batch processing and streaming capabilities.

### Usage

1. **Load Mimi Model**: Download pre-trained weights and load the Mimi model.
2. **Encode Audio**: Convert audio waveforms (24kHz) into codes using `mimi.encode()`.
3. **Decode Codes**: Convert codes back into audio waveforms using `mimi.decode()`.
4. **Streaming Encoding**: Process audio in frames for streaming applications using `mimi.streaming()`.

### Code Example
```python
from huggingface_hub import hf_hub_download
import torch

from moshi.models import loaders, LMGen

# Load Mimi model
mimi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MIMI_NAME)
mimi = loaders.get_mimi(mimi_weight, device='cpu')
mimi.set_num_codebooks(8)  # Set number of codebooks (up to 32, limited to 8 for Moshi)

# Prepare audio (ensure it's 24kHz, [B, C=1, T])
wav = torch.randn(1, 1, 24000 * 10)  

# Encode audio to codes
with torch.no_grad():
    codes = mimi.encode(wav)  # Output shape: [B, K = 8, T]

# Decode codes back to audio
with torch.no_grad():
    decoded = mimi.decode(codes)

# Streaming encoding example
frame_size = mimi.frame_size
all_codes = []
with mimi.streaming(batch_size=1):
    for offset in range(0, wav.shape[-1], frame_size):
        frame = wav[:, :, offset: offset + frame_size]
        codes = mimi.encode(frame) # Output shape: [B, K = 8, 1]
        all_codes.append(codes)

## WARNING: When streaming, make sure to always feed a total amount of audio that is a multiple
#           of the frame size (1920). You should pad or buffer accordingly. Since version 0.2.5a,
            Mimi no longer supports partial frames in streaming mode. Besides, when executing on GPU,
            you should always pass the same amount of audio, as the calls are CUDAGraphed for efficiency.
```

### Parameters

- **`wav`** (torch.Tensor): Input audio waveform tensor. Expected shape `[B, C=1, T]`, with a sample rate of 24kHz.
- **`device`** (str): Device to load the model on ('cpu' or 'cuda').
- **`num_codebooks`** (int): Number of codebooks to use for encoding. Limited to 8 for Moshi.
- **`batch_size`** (int): Batch size for streaming operations.

### Response

- **`codes`** (torch.Tensor): Encoded audio codes. Shape `[B, K, T]` for batch encoding, `[B, K, 1]` for streaming frames.
- **`decoded`** (torch.Tensor): Decoded audio waveform tensor.

```

--------------------------------

### Test Mimi Streaming

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

This command executes a test script for Mimi's streaming capabilities. Ensure you are in the root directory of the repository and have downloaded the necessary asset.

```bash
wget https://github.com/metavoiceio/metavoice-src/raw/main/assets/bria.mp3
python scripts/mimi_streaming_test.py
```

--------------------------------

### Load and Use Moshi LM for Generation

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

Load the Moshi language model on a GPU for text generation. Initialize LMGen with the model and sampling parameters.

```python
# Now if you have a GPU around.
mimi.cuda()
moshi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MOSHI_NAME)
moshi = loaders.get_moshi_lm(moshi_weight, device='cuda')
lm_gen = LMGen(moshi, temp=0.8, temp_text=0.7)  # this handles sampling params etc.
out_wav_chunks = []
```

--------------------------------

### loaders.CheckpointInfo.from_hf_repo

Source: https://context7.com/kyutai-labs/moshi/llms.txt

A high-level factory method to download all necessary model artifacts (LM weights, Mimi weights, tokenizer, and optional LoRA weights) from a Hugging Face repository and bundle them into a `CheckpointInfo` object.

```APIDOC
## loaders.CheckpointInfo.from_hf_repo

High-level factory that downloads all model artifacts (LM weights, Mimi weights, SentencePiece tokenizer, and optional LoRA weights) from a Hugging Face repository and returns a `CheckpointInfo` bundle.

### Usage
```python
from moshi.models import loaders
import torch

# Load Moshika (female voice)
checkpoint = loaders.CheckpointInfo.from_hf_repo("kyutai/moshika-pytorch-bf16")

mimi = checkpoint.get_mimi(device='cuda')
lm   = checkpoint.get_moshi(device='cuda', dtype=torch.bfloat16)
tokenizer = checkpoint.get_text_tokenizer()  # SentencePieceProcessor

print(checkpoint.model_type)   # "moshi"
print(checkpoint.lm_gen_config)

# Load quantized int8 model (Rust/Candle only)
checkpoint_q8 = loaders.CheckpointInfo.from_hf_repo("kyutai/moshiko-candle-q8")

# Override specific weight files (e.g., point to a local fine-tune)
checkpoint_custom = loaders.CheckpointInfo.from_hf_repo(
    "kyutai/moshika-pytorch-bf16",
    moshi_weights="file:///path/to/my/finetuned_model.safetensors",
)
```
```

--------------------------------

### Run Moshi Benchmark

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

This command runs the benchmark script for Moshi. A GPU is required for this test.

```bash
python scripts/moshi_benchmark.py
```

--------------------------------

### Load Mimi Audio Codec Programmatically

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Load and instantiate the Mimi audio codec using `loaders.get_mimi` for encoding/decoding audio tensors. The codec operates at 24 kHz and is configured for 8 active codebooks for Moshi compatibility. Batch encode/decode operations require audio tensors of shape [B, C=1, T] and operate on 24kHz mono WAV data.

```python
from huggingface_hub import hf_hub_download
import torch
from moshi.models import loaders

# Download and load Mimi codec
mimi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MIMI_NAME)
mimi = loaders.get_mimi(mimi_weight, device='cpu')
mimi.set_num_codebooks(8)  # 8 codebooks for Moshi; up to 32 supported

# Batch encode/decode — wav must be 24kHz mono, shape [B, C=1, T]
wav = torch.randn(1, 1, 24000 * 10)  # 10 seconds of audio
with torch.no_grad():
    codes = mimi.encode(wav)       # → [B=1, K=8, T=125]
    decoded = mimi.decode(codes)   # → [B=1, C=1, T≈240000]

print(f"Encoded shape: {codes.shape}")     # torch.Size([1, 8, 125])
print(f"Decoded shape: {decoded.shape}")   # torch.Size([1, 1, 240000])

# Load via CheckpointInfo (also fetches LM and tokenizer)
checkpoint = loaders.CheckpointInfo.from_hf_repo("kyutai/moshika-pytorch-bf16")
mimi2 = checkpoint.get_mimi(device='cuda')
```

--------------------------------

### Generate HTTPS Certificate and Key

Source: https://github.com/kyutai-labs/moshi/blob/main/FAQ.md

Use this command to generate self-signed SSL certificate and private key files for serving over HTTPS. These are valid for 365 days and configured for localhost.

```bash
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=localhost"
```

--------------------------------

### Load and Use Mimi Model (Non-Streaming)

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

Programmatically load the Mimi model for audio encoding and decoding. Ensure input WAV is 24kHz; resample if necessary. The model supports streaming, but non-streaming usage is shown here.

```python
from huggingface_hub import hf_hub_download
import torch

from moshi.models import loaders, LMGen

mimi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MIMI_NAME)
mimi = loaders.get_mimi(mimi_weight, device='cpu')
mimi.set_num_codebooks(8)  # up to 32 for mimi, but limited to 8 for moshi.

# wav should be 24kHz, if not, resample using for instance torchaudio.functional.resample
wav = torch.randn(1, 1, 24000 * 10)  # should be [B, C=1, T]

with torch.no_grad():
    codes = mimi.encode(wav)  # [B, K = 8, T]
    decoded = mimi.decode(codes)
```

--------------------------------

### Moshi Backend Configuration

Source: https://context7.com/kyutai-labs/moshi/llms.txt

JSON configuration file for the Rust backend, specifying model paths, ports, and directories.

```json
// moshi-backend/config.json
{
  "instance_name": "my-moshi",
  "hf_repo": "kyutai/moshiko-candle-bf16",
  "lm_model_file": "$HOME/tmp/model.safetensors",
  "text_tokenizer_file": "$HOME/tmp/tokenizer_spm_32k_3.model",
  "mimi_model_file": "$HOME/tmp/tokenizer-e351c8d8-checkpoint125.safetensors",
  "mimi_num_codebooks": 8,
  "static_dir": "../client/dist",
  "addr": "0.0.0.0",
  "port": 8998,
  "cert_dir": ".",
  "log_dir": "$HOME/tmp/moshi-logs"
}
```

--------------------------------

### Load Moshi LM and Run Token-Level Streaming Inference

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Load the Moshi language model and perform autoregressive generation. This process involves encoding user audio into Mimi codes and then using the language model to generate text and audio tokens, which are subsequently decoded into audio chunks.

```python
import torch
from huggingface_hub import hf_hub_download
from moshi.models import loaders, LMGen

# Load Mimi + Moshi on GPU
mimi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MIMI_NAME)
moshi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MOSHI_NAME)

mimi = loaders.get_mimi(mimi_weight, device='cuda')
mimi.set_num_codebooks(8)

moshi_lm = loaders.get_moshi_lm(moshi_weight, device='cuda')
lm_gen = LMGen(moshi_lm, temp=0.8, temp_text=0.7)

# Encode user audio into codes
wav = torch.randn(1, 1, 24000 * 3).cuda()  # 3s of user audio
frame_size = mimi.frame_size
all_codes = []
with torch.no_grad(), mimi.streaming(1):
    for offset in range(0, wav.shape[-1], frame_size):
        codes = mimi.encode(wav[:, :, offset: offset + frame_size])
        all_codes.append(codes)

# Run Moshi LM + decode output audio
out_wav_chunks = []
with torch.no_grad(), lm_gen.streaming(1), mimi.streaming(1):
    for code in all_codes:
        tokens_out = lm_gen.step(code.cuda())
        # tokens_out shape: [B=1, 1+dep_q=9, T=1]
        # tokens_out[:, 0] = text token (inner monologue)
        # tokens_out[:, 1:] = 8 audio codebook tokens
        if tokens_out is not None:
            wav_chunk = mimi.decode(tokens_out[:, 1:])
            out_wav_chunks.append(wav_chunk)

out_wav = torch.cat(out_wav_chunks, dim=-1)
print(f"Generated audio: {out_wav.shape}")  # [1, 1, ~72000]
```

--------------------------------

### Streaming Execution with Mimi and LM Generation

Source: https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md

This snippet demonstrates streaming audio generation using Mimi and language model generation (lm_gen). It processes codes, decodes tokens into audio chunks, and concatenates them.

```python
with torch.no_grad(), lm_gen.streaming(1), mimi.streaming(1):
    for idx, code in enumerate(all_codes):
        tokens_out = lm_gen.step(code.cuda())
        # tokens_out is [B, 1 + 8, 1], with tokens_out[:, 1] representing the text token.
        if tokens_out is not None:
            wav_chunk = mimi.decode(tokens_out[:, 1:])
            out_wav_chunks.append(wav_chunk)
        print(idx, end='\r')
out_wav = torch.cat(out_wav_chunks, dim=-1)
```

--------------------------------

### Load Mimi Audio Codec Programmatically

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Load and instantiate the Mimi audio codec programmatically for encoding/decoding audio tensors. Mimi operates at 24 kHz with 8 active codebooks for Moshi compatibility.

```APIDOC
## Load Mimi Audio Codec Programmatically

### Description
Load and instantiate the Mimi audio codec programmatically for encoding/decoding audio tensors. Mimi operates at 24 kHz with 8 active codebooks for Moshi compatibility (up to 32 supported).

### Usage
```python
from huggingface_hub import hf_hub_download
import torch
from moshi.models import loaders

# Download and load Mimi codec
mimi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MIMI_NAME)
mimi = loaders.get_mimi(mimi_weight, device='cpu')
mimi.set_num_codebooks(8)  # 8 codebooks for Moshi; up to 32 supported

# Batch encode/decode — wav must be 24kHz mono, shape [B, C=1, T]
wav = torch.randn(1, 1, 24000 * 10)  # 10 seconds of audio
with torch.no_grad():
    codes = mimi.encode(wav)       # → [B=1, K=8, T=125]
    decoded = mimi.decode(codes)   # → [B=1, C=1, T≈240000]

print(f"Encoded shape: {codes.shape}")     # torch.Size([1, 8, 125])
print(f"Decoded shape: {decoded.shape}")   # torch.Size([1, 1, 240000])

# Load via CheckpointInfo (also fetches LM and tokenizer)
checkpoint = loaders.CheckpointInfo.from_hf_repo("kyutai/moshika-pytorch-bf16")
mimi2 = checkpoint.get_mimi(device='cuda')
```

### Functions
- `loaders.get_mimi(weight_path, device)`: Loads the Mimi codec from a given weight path and device.
- `mimi.set_num_codebooks(num_codebooks)`: Sets the number of active codebooks for the codec.
- `mimi.encode(wav)`: Encodes audio tensors into codes.
- `mimi.decode(codes)`: Decodes codes back into audio tensors.

### Classes
- `loaders.CheckpointInfo`:
  - `from_hf_repo(repo_id)`: Loads checkpoint information, including LM and tokenizer, from a Hugging Face repository.
  - `get_mimi(device)`: Retrieves the Mimi codec instance associated with the checkpoint.
```

--------------------------------

### Run MLX Local Inference (Quantized)

Source: https://github.com/kyutai-labs/moshi/blob/main/README.md

Executes local inference using the MLX implementation on macOS. Supports weights quantized to 4 or 8 bits. Use --hf-repo to specify a different model.

```bash
python -m moshi_mlx.local -q 4   # weights quantized to 4 bits
```

```bash
python -m moshi_mlx.local -q 8   # weights quantized to 8 bits
```

```bash
python -m moshi_mlx.local -q 4 --hf-repo kyutai/moshika-mlx-q4
```

```bash
python -m moshi_mlx.local -q 8 --hf-repo kyutai/moshika-mlx-q8
```

--------------------------------

### Run Moshi Rust Inference Server

Source: https://github.com/kyutai-labs/moshi/blob/main/rust/README.md

Execute the Moshi inference server using Cargo. Supports CUDA or Metal features and different configuration files.

```bash
cargo run --features cuda --bin moshi-backend -r -- --config moshi-backend/config.json standalone
```

--------------------------------

### loaders.get_moshi_lm / LMGen

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Load the Moshi 7B language model and run token-level streaming inference. `LMGen` manages sampling parameters and the autoregressive generation loop, consuming Mimi codes and producing text and audio tokens.

```APIDOC
## loaders.get_moshi_lm / LMGen

Load the Moshi 7B language model and run token-level streaming inference. `LMGen` wraps sampling parameters and manages the autoregressive generation loop. Each step consumes one frame of Mimi codes and produces text + audio tokens.

### Usage
```python
import torch
from huggingface_hub import hf_hub_download
from moshi.models import loaders, LMGen

# Load Mimi + Moshi on GPU
mimi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MIMI_NAME)
moshi_weight = hf_hub_download(loaders.DEFAULT_REPO, loaders.MOSHI_NAME)

mimi = loaders.get_mimi(mimi_weight, device='cuda')
mimi.set_num_codebooks(8)

moshi_lm = loaders.get_moshi_lm(moshi_weight, device='cuda')
lm_gen = LMGen(moshi_lm, temp=0.8, temp_text=0.7)

# Encode user audio into codes
wav = torch.randn(1, 1, 24000 * 3).cuda()  # 3s of user audio
frame_size = mimi.frame_size
all_codes = []
with torch.no_grad(), mimi.streaming(1):
    for offset in range(0, wav.shape[-1], frame_size):
        codes = mimi.encode(wav[:, :, offset: offset + frame_size])
        all_codes.append(codes)

# Run Moshi LM + decode output audio
out_wav_chunks = []
with torch.no_grad(), lm_gen.streaming(1), mimi.streaming(1):
    for code in all_codes:
        tokens_out = lm_gen.step(code.cuda())
        # tokens_out shape: [B=1, 1+dep_q=9, T=1]
        # tokens_out[:, 0] = text token (inner monologue)
        # tokens_out[:, 1:] = 8 audio codebook tokens
        if tokens_out is not None:
            wav_chunk = mimi.decode(tokens_out[:, 1:])
            out_wav_chunks.append(wav_chunk)

out_wav = torch.cat(out_wav_chunks, dim=-1)
print(f"Generated audio: {out_wav.shape}")  # [1, 1, ~72000]
```
```

--------------------------------

### Run Batch Text-to-Speech Pipeline

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Execute a batch text-to-speech pipeline using the `moshi.run_tts` command. This script processes a JSONL file containing text and voice requests, generating WAV files and associated debug artifacts in a specified output folder.

```bash
# Install, then run TTS
python -m moshi.run_tts \
    --out-folder ./tts-outputs \
    --hf-repo kyutai/tts-1b \
    --voice-repo kyutai/tts-voices \
    --batch-size 32 \
    --temp 0.6 \
    --cfg-coef 2.0 \
    requests.jsonl
```

```jsonl
{"turns": ["Hello, welcome to Moshi!", "Nice to meet you."], "voices": ["main_voice", "second_voice"], "id": "sample_001"}
{"turns": ["The weather today is sunny."], "voices": ["main_voice"], "id": "sample_002"}
```

```python
# Expected outputs in ./tts-outputs/
# sample_001.wav   — generated audio
# sample_001.safetensors  — raw token tensors (debug)
# sample_001.json  — generation metadata
```

--------------------------------

### Generate TLS Certificate for Client Dev Server

Source: https://context7.com/kyutai-labs/moshi/llms.txt

Generates a self-signed TLS certificate and private key for local development. Ensure the generated key.pem and cert.pem files are copied to the client directory.

```bash
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
    -days 365 -nodes -subj "/CN=localhost"
```