### Install Package

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Installs the moshi-finetune package in editable mode. This makes the project's code available for use.

```python
%pip install -e /content/moshi-finetune
```

--------------------------------

### Run Training on a Single GPU

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Use this command to start the training process on a single GPU. Ensure you are using `torchrun` even for single-GPU setups.

```sh
torchrun --nproc-per-node 1 -m train example/moshi_7B.yaml
```

--------------------------------

### Start Training

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Initiates the training process using `torchrun` with the specified configuration file. Ensure the `run_dir` in the config is not already in use or has been cleared.

```bash
# start training

!cd /content/moshi-finetune && torchrun --nproc-per-node 1 -m train /content/example.yaml
```

--------------------------------

### Install Dependencies with Pip

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Install the project dependencies using pip after navigating into the cloned repository. A virtual environment is recommended.

```sh
cd moshi-finetune
pip install -e .
```

--------------------------------

### Install Gradio

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Installs the Gradio library, which is used for creating user interfaces for machine learning models. This is required for the inference server.

```bash
!pip install gradio
```

--------------------------------

### Start Inference Server

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Launches the Moshi inference server using Gradio for tunneling audio data. Specify the path to the LoRA weights and the configuration file from the training output.

```bash
!python -m moshi.server --gradio-tunnel --lora-weight=/content/test/checkpoints/checkpoint_000300/consolidated/lora.safetensors --config-path=/content/test/checkpoints/checkpoint_000300/consolidated/config.json
```

--------------------------------

### Install Moshi for Inference

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Install the Moshi package for inference if it's not already available in your environment. This command installs directly from the Git repository.

```sh
pip install git+https://git@github.com/kyutai-labs/moshi.git#egg=moshi&subdirectory=moshi
```

--------------------------------

### Launch Training with uv

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

If using `uv` for environment management, you can prepend `uv run` to your `torchrun` command. This is useful for projects where dependencies are managed without a global `pip install`.

```bash
uv run torchrun --nproc-per-node 8 --master_port $RANDOM -m train example/moshi_7B.yaml
```

--------------------------------

### Accessing Inference Server

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

After starting the Moshi inference server, access it via a web browser to interact with the fine-tuned model.

```bash
# Then open http://localhost:8998 in a browser to chat with your fine-tuned Moshi
```

--------------------------------

### Inference with LoRA Adapter

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Starts the Moshi inference server using only the LoRA adapter weights. This is a lightweight option that applies the adapter on top of the base model.

```bash
CHECKPOINT_DIR="$HOME/my_run/checkpoints/checkpoint_002000/consolidated"

# LoRA adapter inference (lightweight — applies adapter on top of base model)
python -m moshi.server \
    --lora-weight="$CHECKPOINT_DIR/lora.safetensors" \
    --config-path="$CHECKPOINT_DIR/config.json"
```

--------------------------------

### Annotate Dataset Locally

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Use this command to annotate your dataset locally using a single GPU and English language model. Ensure you have the necessary dependencies installed.

```bash
python annotate.py data/mycooldataset.jsonl \
    --lang en \
    --whisper_model medium \
    --local
```

--------------------------------

### Inference with Full Fine-Tuned Model

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Starts the Moshi inference server using the fully fine-tuned or merged model weights. This option loads the complete model weights.

```bash
CHECKPOINT_DIR="$HOME/my_run/checkpoints/checkpoint_002000/consolidated"

# Full fine-tuned / merged model inference
python -m moshi.server \
    --moshi-weight="$CHECKPOINT_DIR/consolidated.safetensors" \
    --config-path="$CHECKPOINT_DIR/config.json"
```

--------------------------------

### Expected Annotation Output Structure

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

The annotation process generates a JSON file for each audio file, containing an 'alignments' list. Each item in the list includes the transcribed word, its start and end timestamps in seconds, and the speaker identifier.

```json
{
  "alignments": [
    ["Hello,", [0.12, 0.45], "SPEAKER_MAIN"],
    ["how",    [0.50, 0.65], "SPEAKER_MAIN"],
    ...
  ]
}
```

--------------------------------

### Initialize LoRA Parameters

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Initializes LoRA adapter weights allocated on the meta device. 'lora_A' matrices are initialized with Kaiming uniform, and 'lora_B' matrices with zeros, matching the original LoRA paper.

```python
import torch
from finetune.wrapped_model import initialize_lora_parameters

# After model is created on meta device and base weights loaded on rank 0:
initialize_lora_parameters(model, param_dtype=torch.bfloat16)
# lora_A layers: kaiming_uniform initialised
# lora_B layers: zero initialised → net effect = identity at step 0
```

--------------------------------

### LoRA Initialisation

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Initialises LoRA adapter weights that were allocated on the meta device: `lora_A` matrices with Kaiming uniform, `lora_B` matrices with zeros, matching the original LoRA paper.

```APIDOC
## LoRA Initialisation — `initialize_lora_parameters(model, param_dtype)`

Initialises LoRA adapter weights that were allocated on the meta device: `lora_A` matrices with Kaiming uniform, `lora_B` matrices with zeros, matching the original LoRA paper (Hu et al., 2021).

```python
import torch
from finetune.wrapped_model import initialize_lora_parameters

# After model is created on meta device and base weights loaded on rank 0:
initialize_lora_parameters(model, param_dtype=torch.bfloat16)
# lora_A layers: kaiming_uniform initialised
# lora_B layers: zero initialised → net effect = identity at step 0
```
```

--------------------------------

### FSDP Model Initialisation

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Loads the Moshi LMModel in LoRA or full fine-tuning mode, shards it across all available GPUs using PyTorch FSDP with FULL_SHARD strategy, and returns the wrapped model ready for training.

```APIDOC
## FSDP Model Initialisation — `get_fsdp_model(args, checkpoint_info)`

Loads the Moshi `LMModel` in LoRA or full fine-tuning mode, shards it across all available GPUs using PyTorch FSDP with `FULL_SHARD` strategy, and returns the wrapped model ready for training. On rank 0 the weights are read from SafeTensors; other ranks use a `param_init_fn` to materialise from meta tensors.

```python
from moshi.models import loaders
from finetune.args import TrainArgs
from finetune.wrapped_model import get_fsdp_model

args = TrainArgs.load("example/moshi_7B.yaml", drop_extra_fields=False)

checkpoint_info = loaders.CheckpointInfo.from_hf_repo(
    hf_repo="kyutai/moshiko-pytorch-bf16"
)

# Returns FullyShardedDataParallel when world_size > 1, else LMModel on CUDA
model = get_fsdp_model(args, checkpoint_info)
# → "7,000,000,000 out of 7,000,000,000 parameters: 0.18% are finetuned (LoRA)"
```
```

--------------------------------

### Clone Repository

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Clones the moshi-finetune repository to the local environment. This is the first step to set up the project.

```python
%cd /content/
!git clone https://github.com/kyutai-labs/moshi-finetune.git
```

--------------------------------

### Build Dataset with Mixed Sources and Weights

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Builds a dataset by combining multiple data sources with specified weights. Ensure the 'tokenizer' object is properly initialized before use.

```python
iterator = build_dataset(
    pretrain_data="data/daily-talk.jsonl:0.7,data/custom.jsonl:0.3",
    instruct_tokenizer=tokenizer,
    seed=42,
    rank=0,
    world_size=1,
    is_eval=False,
    shuffle_pretrain=True,
)

sample = next(iterator)
print(sample.codes.shape)  # [1, 9, num_frames]
```

--------------------------------

### Clone Moshi-Finetune Repository

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Use this command to clone the project repository to your local machine.

```sh
git clone git@github.com:kyutai-labs/moshi-finetune.git
```

--------------------------------

### Run Training on Multiple GPUs (8)

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Execute this command to initiate training across multiple GPUs. The `--master_port` is set to a random port for flexibility.

```sh
torchrun --nproc-per-node 8 --master_port $RANDOM -m train example/moshi_7B.yaml
```

--------------------------------

### Initialize FSDP Model for Moshi

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Loads the Moshi LMModel in LoRA or full fine-tuning mode and shards it across GPUs using PyTorch FSDP. Weights are read from SafeTensors on rank 0, and other ranks materialize from meta tensors.

```python
from moshi.models import loaders
from finetune.args import TrainArgs
from finetune.wrapped_model import get_fsdp_model

args = TrainArgs.load("example/moshi_7B.yaml", drop_extra_fields=False)

checkpoint_info = loaders.CheckpointInfo.from_hf_repo(
    hf_repo="kyutai/moshiko-pytorch-bf16"
)

# Returns FullyShardedDataParallel when world_size > 1, else LMModel on CUDA
model = get_fsdp_model(args, checkpoint_info)
# → "7,000,000,000 out of 7,000,000,000 parameters: 0.18% are finetuned (LoRA)"
```

--------------------------------

### Build Finetuning Dataset

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Creates an iterator over Sample objects from weighted JSONL data sources. Supports shuffling per epoch and shard-aware loading for distributed training.

```python
from finetune.data.dataset import build_dataset

# Single source
iterator = build_dataset(
    pretrain_data="data/mycooldataset.jsonl",
    instruct_tokenizer=tokenizer,
    seed=42,
    rank=0,
    world_size=1,
    is_eval=False,
    shuffle_pretrain=True,
)

```

--------------------------------

### Moshi Finetune Training Configuration

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

This YAML file configures training parameters for Moshi models, including data paths, model repository, LoRA settings, optimizer parameters, and checkpointing options. Adjust these settings based on your dataset and hardware.

```yaml
# example/moshi_7B.yaml — recommended quick-start config
data:
  train_data: 'data/mycooldataset.jsonl'   # required
  eval_data:  ''                            # optional
  shuffle:    true

moshi_paths:
  hf_repo_id: "kyutai/moshiko-pytorch-bf16"  # loads weights from HF Hub

full_finetuning: false   # use LoRA, not full fine-tuning
lora:
  enable:   true
  rank:     128           # adapter rank; keep ≤128 for efficiency
  scaling:  2.0
  ft_embed: false         # also fine-tune embedding matrices?

first_codebook_weight_multiplier: 100.0  # upweight semantic codebook loss
text_padding_weight: 0.5                 # downweight padding token loss

duration_sec: 100     # max sequence length in seconds
batch_size:   16      # per-GPU batch size
max_steps:    2000
gradient_checkpointing: true

optim:
  lr:          2.0e-6
  weight_decay: 0.1
  pct_start:   0.05   # fraction of steps for LR warm-up

seed:      0
log_freq:  1
eval_freq: 100
do_eval:   false
do_ckpt:   true
ckpt_freq: 100
save_adapters: true   # save only LoRA weights (smaller, compatible with inference)
run_dir: "/path/to/my_run"

# Optional W&B integration
# wandb:
#   project:  "moshi-experiments"
#   run_name: "dailytalk-lora-r128"
#   key:      "<WANDB_API_API_KEY>"
#   offline:  false

```

--------------------------------

### Inference with Fine-Tuned Model

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Demonstrates how to run inference using a fine-tuned Moshi model, with options for using LoRA adapters or a fully merged model.

```APIDOC
## Inference with Fine-Tuned Model

After training, the saved checkpoint is used directly with the `moshi` inference server.

```bash
CHECKPOINT_DIR="$HOME/my_run/checkpoints/checkpoint_002000/consolidated"

# LoRA adapter inference (lightweight — applies adapter on top of base model)
python -m moshi.server \
    --lora-weight="$CHECKPOINT_DIR/lora.safetensors" \
    --config-path="$CHECKPOINT_DIR/config.json"

# Full fine-tuned / merged model inference
python -m moshi.server \
    --moshi-weight="$CHECKPOINT_DIR/consolidated.safetensors" \
    --config-path="$CHECKPOINT_DIR/config.json"

# Then open http://localhost:8998 in a browser to chat with your fine-tuned Moshi
```
```

--------------------------------

### Download Dataset

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Downloads the DailyTalkContiguous dataset using `snapshot_download` from Hugging Face Hub and saves it to a specified local directory. Ensure the target directory exists.

```python
from pathlib import Path

from huggingface_hub import snapshot_download

Path("/content/data/daily-talk-contiguous").mkdir(parents=True, exist_ok=True)

# Download the dataset
local_dir = snapshot_download(
    "kyutai/DailyTalkContiguous",
    repo_type="dataset",
    local_dir="/content/data/daily-talk-contiguous",
)
```

--------------------------------

### Run Inference with Full Fine-tuned Model

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Use this command to run the Moshi web app if you performed full fine-tuning or did not checkpoint LORA only. This command loads weights that do not reference a base model.

```sh
python -m moshi.server \
--moshi-weight=$CHECKPOINT_DIR/consolidated/consolidated.safetensors \
--config-path=$CHECKPOINT_DIR/consolidated/consolidated/config.json 
```

--------------------------------

### Dataset Building

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Creates a (potentially infinite) iterator over `Sample` objects from one or more weighted JSONL data sources. Supports shuffling per epoch and shard-aware loading for distributed training.

```APIDOC
## Dataset Building — `build_dataset(pretrain_data, ...)`

Creates a (potentially infinite) iterator over `Sample` objects from one or more weighted JSONL data sources. Sources can be files or directories; multiple sources are mixed according to their weights. Supports shuffling per epoch and shard-aware loading for distributed training.

```python
from finetune.data.dataset import build_dataset

# Single source
iterator = build_dataset(
    pretrain_data="data/mycooldataset.jsonl",
    instruct_tokenizer=tokenizer,
    seed=42,
    rank=0,
    world_size=1,
    is_eval=False,
    shuffle_pretrain=True,
)

```
```

--------------------------------

### Run Inference with LORA Fine-tuned Model

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Use this command to run the Moshi web app when you have trained using LORA. It applies the LORA adapter on top of the base model's weights.

```sh
python -m moshi.server \
  --lora-weight=$CHECKPOINT_DIR/consolidated/lora.safetensors \
  --config-path=$CHECKPOINT_DIR/consolidated/config.json
```

--------------------------------

### Define Training Configuration

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Defines the training configuration using a YAML string, including data paths, model settings, LoRA parameters, and training hyperparameters. This configuration is then saved to a local YAML file.

```python
# define training configuration
# for your own use cases, you might want to change the data paths, model path, run_dir, and other hyperparameters
import yaml

config = """
# data
data:
  train_data: '/content/data/daily-talk-contiguous/dailytalk.jsonl' # Fill
  eval_data: '' # Optionally Fill
  shuffle: true

# model
moshi_paths:
  hf_repo_id: "kyutai/moshiko-pytorch-bf16"


full_finetuning: false # Activate lora.enable if partial finetuning
lora:
  enable: true
  rank: 128
  scaling: 2.
  ft_embed: false

# training hyperparameters
first_codebook_weight_multiplier: 100.
text_padding_weight: .5


# tokens per training steps = batch_size x num_GPUs x duration_sec
# we recommend a sequence duration of 300 seconds
# If you run into memory error, you can try reduce the sequence length
duration_sec: 100
batch_size: 1
max_steps: 300

gradient_checkpointing: true # Activate checkpointing of layers

# optim
optim:
  lr: 2.e-6
  weight_decay: 0.1
  pct_start: 0.05

# other
seed: 0
log_freq: 10
eval_freq: 1
do_eval: False
ckpt_freq: 10

save_adapters: True

run_dir: "/content/test"  # Fill
"""

# save the same file locally into the example.yaml file
with open("/content/example.yaml", "w") as file:
    yaml.dump(yaml.safe_load(config), file)
```

--------------------------------

### Generate Dataset JSONL File

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Python script to generate a .jsonl file for the dataset by collecting paths and durations of .wav files in a directory. Requires the 'sphn' library.

```python
import sphn
import json
from pathlib import Path

paths = [str(f) for f in Path("wav-dir").glob("*.wav")]
durations = sphn.durations(paths)

with open("data.jsonl", "w") as fobj:
    for p, d in zip(paths, durations):
        if d is None:
            continue
        json.dump({"path": p, "duration": d}, fobj)
        fobj.write("\n")
```

--------------------------------

### Annotate Dataset with JSON Transcripts

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/README.md

Use the 'annotate.py' script to generate .json transcripts for audio files. This script can also be run in a distributed manner with SLURM.

```sh
python annotate.py {your jsonl file}
```

--------------------------------

### Annotate Dataset on SLURM

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

For distributed annotation on a SLURM cluster, specify the number of shards, partition, and log folder. This command demonstrates setting up 64 parallel annotation shards.

```bash
python annotate.py data/mycooldataset.jsonl \
    --lang en \
    --whisper_model medium \
    --shards 64 \
    --partition gpu_partition \
    --log_folder ~/tmp/annotate_logs
```

--------------------------------

### Programmatic Training Invocation

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

You can also invoke the training function directly from a Python script by importing the `train` function and passing the configuration file path as an argument.

```python
from train import train
train("example/moshi_7B.yaml")
```

--------------------------------

### Remove Previous Run Directory (Optional)

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Removes the run directory if it exists. This command should only be used if a previous training run created this directory and needs to be cleared before a new run.

```bash
# make sure the run_dir has not been created before
# only run this when you ran torchrun previously and created the /content/test_ultra file
# ! rm -r /content/test
```

--------------------------------

### Set CUDA Environment Variables

Source: https://github.com/kyutai-labs/moshi-finetune/blob/main/tutorials/moshi_finetune.ipynb

Configures CUDA environment variables to specify the GPU device for training. Ensure CUDA is available and the specified device index is valid.

```python
# these info is needed for training
import os

os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
```

--------------------------------

### Checkpointing - save_checkpoint

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Saves the model state to a specified directory. Supports saving only LoRA adapters or the full model. Automatically manages the number of checkpoints to keep.

```APIDOC
## Checkpointing — `Checkpointer.save_checkpoint(save_only_lora, dtype)`

Saves model state to `{run_dir}/checkpoints/checkpoint_{step:06d}/consolidated/` as a SafeTensors file. In LoRA mode with `save_only_lora=True`, only adapter weights are saved (small files, compatible with Moshi inference server). With `save_only_lora=False`, LoRA weights are merged into the base model before saving. Automatically rotates old checkpoints, keeping at most `num_ckpt_keep` recent ones.

```python
from pathlib import Path
from finetune.checkpointing import Checkpointer
from finetune.utils import TrainState

state = TrainState(max_steps=2000)
state.step = 500

checkpointer = Checkpointer(
    model=model,
    state=state,
    config=lm_config,
    run_dir=Path("/path/to/my_run"),
    optimizer=optimizer,
    num_ckpt_keep=3,
    full_finetuning=False,
)

# Save only LoRA adapters at step 500
checkpointer.save_checkpoint(save_only_lora=True, dtype=torch.bfloat16)
# Creates: /path/to/my_run/checkpoints/checkpoint_000500/consolidated/lora.safetensors
#          /path/to/my_run/checkpoints/checkpoint_000500/consolidated/config.json
```
```

--------------------------------

### Evaluation - evaluate

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Runs the model in evaluation mode, calculating and accumulating cross-entropy losses. Results are stored in the `TrainState` and are useful for monitoring training progress.

```APIDOC
## Evaluation — `evaluate(model, eval_data_loader, state, args)`

Runs the model in eval mode over up to 40 batches (divided by world size), accumulates text and audio cross-entropy losses across all ranks via `all_reduce`, and writes `this_eval_loss`, `this_eval_perplexity`, `this_audio_loss`, and `this_text_loss` to the `TrainState`. Called periodically during training at `eval_freq` steps.

```python
from finetune.eval import evaluate

# Called inside the training loop
evaluate(model, eval_data_loader, state, args)

print(f"Eval loss:        {state.this_eval_loss:.4f}")
print(f"Eval perplexity:  {state.this_eval_perplexity:.2f}")
print(f"Audio loss:       {state.this_audio_loss:.4f}")
print(f"Text loss:        {state.this_text_loss:.4f}")
# Example output:
# Eval loss:       2.1340
# Eval perplexity: 4.45
# Audio loss:      1.8921
# Text loss:       0.2419
```
```

--------------------------------

### Evaluate Model Performance

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Runs the model in evaluation mode, calculating and accumulating cross-entropy losses. The results are stored in the `TrainState` object and are typically printed during the training loop.

```python
from finetune.eval import evaluate

# Called inside the training loop
evaluate(model, eval_data_loader, state, args)

print(f"Eval loss:        {state.this_eval_loss:.4f}")
print(f"Eval perplexity:  {state.this_eval_perplexity:.2f}")
print(f"Audio loss:       {state.this_audio_loss:.4f}")
print(f"Text loss:        {state.this_text_loss:.4f}")
# Example output:
# Eval loss:       2.1340
# Eval perplexity: 4.45
# Audio loss:      1.8921
# Text loss:       0.2419
```

--------------------------------

### Save Model Checkpoint with Checkpointer

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Saves the model state, including LoRA adapters or the full model, to a specified directory. This function automatically manages checkpoint rotation. Use `save_only_lora=True` for smaller adapter-only saves.

```python
from pathlib import Path
from finetune.checkpointing import Checkpointer
from finetune.utils import TrainState

state = TrainState(max_steps=2000)
state.step = 500

checkpointer = Checkpointer(
    model=model,
    state=state,
    config=lm_config,
    run_dir=Path("/path/to/my_run"),
    optimizer=optimizer,
    num_ckpt_keep=3,
    full_finetuning=False,
)

# Save only LoRA adapters at step 500
checkpointer.save_checkpoint(save_only_lora=True, dtype=torch.bfloat16)
# Creates: /path/to/my_run/checkpoints/checkpoint_000500/consolidated/lora.safetensors
#          /path/to/my_run/checkpoints/checkpoint_000500/consolidated/config.json
```

--------------------------------

### Loss Computation

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Computes weighted cross-entropy loss with masking, supporting two modes: "audio" (which optionally up-weights the first semantic codebook) and "text" (which optionally down-weights padding tokens).

```APIDOC
## Loss Computation — `compute_loss_with_mask(logits, target, target_mask, mode, ...)`

Computes weighted cross-entropy loss with masking, supporting two modes: `"audio"` (which optionally up-weights the first semantic codebook via `first_codebook_weight_multiplier`) and `"text"` (which optionally down-weights padding tokens via `text_padding_weight`).

```python
import torch
from finetune.loss import compute_loss_with_mask

# Audio loss — upweight first codebook (semantic tokens)
audio_loss = compute_loss_with_mask(
    logits=output.logits,                          # [B, dep_q, T, vocab]
    target=codes[:, audio_offset : audio_offset + dep_q],
    target_mask=output.mask,                       # bool tensor
    mode="audio",
    first_codebook_weight_multiplier=100.0,
)

# Text loss — downweight padding tokens
text_loss = compute_loss_with_mask(
    logits=output.text_logits,                     # [B, 1, T, text_vocab]
    target=codes[:, :audio_offset],
    target_mask=output.text_mask,
    mode="text",
    text_padding_weight=0.5,
    text_padding_ids={model.text_padding_token_id, model.end_of_text_padding_id},
)

total_loss = text_loss + audio_loss
total_loss.backward()
```
```

--------------------------------

### Audio–Text Interleaving

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Encodes a stereo audio waveform with Mimi, reads the corresponding word-alignment .json, and interleaves audio codes with time-aligned text tokens into a single `codes` tensor.

```APIDOC
## Audio–Text Interleaving — `InterleavedTokenizer.__call__(wav, start_sec, path)`

Encodes a stereo audio waveform with Mimi, reads the corresponding word-alignment `.json`, and interleaves audio codes with time-aligned text tokens into a single `codes` tensor consumed by the model. Handles padding, chunking to `duration_sec`, and speaker filtering.

```python
import torch
import sphn
from finetune.data.interleaver import Interleaver, InterleavedTokenizer
from moshi.models import loaders

checkpoint_info = loaders.CheckpointInfo.from_hf_repo("kyutai/moshiko-pytorch-bf16")
mimi = checkpoint_info.get_mimi(device="cuda")
spm  = checkpoint_info.get_text_tokenizer()

interleaver = Interleaver(
    spm,
    audio_frame_rate=mimi.frame_rate,   # 12.5 Hz
    text_padding=model.text_padding_token_id,
    end_of_text_padding=model.end_of_text_padding_id,
    zero_padding=model.zero_token_id,
    keep_main_only=True,
)

tokenizer = InterleavedTokenizer(mimi, interleaver, duration_sec=10.0)

wav, sr = sphn.read("data_stereo/a.wav")  # stereo, shape [2, samples]
sample = tokenizer(wav, start_sec=0.0, path="data_stereo/a.wav")
# sample.codes: torch.Tensor of shape [1, 1 + dep_q, num_audio_frames]
# row 0 → text tokens; rows 1..dep_q → audio codebook tokens
print(sample.codes.shape)  # e.g. torch.Size([1, 9, 125])
```
```

--------------------------------

### Compute Loss with Masking

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Computes weighted cross-entropy loss with masking. Supports 'audio' mode with optional up-weighting of the first semantic codebook and 'text' mode with optional down-weighting of padding tokens.

```python
import torch
from finetune.loss import compute_loss_with_mask

# Audio loss — upweight first codebook (semantic tokens)
audio_loss = compute_loss_with_mask(
    logits=output.logits,                          # [B, dep_q, T, vocab]
    target=codes[:, audio_offset : audio_offset + dep_q],
    target_mask=output.mask,                       # bool tensor
    mode="audio",
    first_codebook_weight_multiplier=100.0,
)

# Text loss — downweight padding tokens
text_loss = compute_loss_with_mask(
    logits=output.text_logits,                     # [B, 1, T, text_vocab]
    target=codes[:, :audio_offset],
    target_mask=output.text_mask,
    mode="text",
    text_padding_weight=0.5,
    text_padding_ids={model.text_padding_token_id, model.end_of_text_padding_id},
)

total_loss = text_loss + audio_loss
total_loss.backward()
```

--------------------------------

### Interleave Audio and Text Tokens

Source: https://context7.com/kyutai-labs/moshi-finetune/llms.txt

Encodes audio, reads word-alignment data, and interleaves audio codes with time-aligned text tokens into a single tensor. Handles padding, chunking, and speaker filtering.

```python
import torch
import sphn
from finetune.data.interleaver import Interleaver, InterleavedTokenizer
from moshi.models import loaders

checkpoint_info = loaders.CheckpointInfo.from_hf_repo("kyutai/moshiko-pytorch-bf16")
mimi = checkpoint_info.get_mimi(device="cuda")
spm  = checkpoint_info.get_text_tokenizer()

interleaver = Interleaver(
    spm,
    audio_frame_rate=mimi.frame_rate,   # 12.5 Hz
    text_padding=model.text_padding_token_id,
    end_of_text_padding=model.end_of_text_padding_id,
    zero_padding=model.zero_token_id,
    keep_main_only=True,
)

tokenizer = InterleavedTokenizer(mimi, interleaver, duration_sec=10.0)

wav, sr = sphn.read("data_stereo/a.wav")  # stereo, shape [2, samples]
sample = tokenizer(wav, start_sec=0.0, path="data_stereo/a.wav")
# sample.codes: torch.Tensor of shape [1, 1 + dep_q, num_audio_frames]
# row 0 → text tokens; rows 1..dep_q → audio codebook tokens
print(sample.codes.shape)  # e.g. torch.Size([1, 9, 125])
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.