### Install Base WhisperKit Tools

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Install the base whisperkittools package using pip. Ensure you are in the root directory of the cloned repository and have a Python virtual environment activated.

```shell
cd WHISPERKIT_ROOT_DIR && pip install -e .
```

--------------------------------

### Install whisperkittools

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Install the base whisperkittools package with conda. Optional extras can be installed for pipelines, evaluation, Android support, or speaker diarization.

```bash
conda create -n whisperkit python=3.11 -y && conda activate whisperkit
git clone https://github.com/argmaxinc/whisperkittools && cd whisperkittools
pip install -e .
```

```bash
pip install -e '.[pipelines]'
```

```bash
pip install -e '.[evals,pipelines]'
```

```bash
pip install -e '.[android]'
```

```bash
pip install -e '.[diarization]'
```

--------------------------------

### Install Evaluation Dependencies

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Install additional dependencies required for model evaluation and pipelines.

```shell
pip install -e '.[evals,pipelines]'
```

--------------------------------

### Install Pipeline Dependencies

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Install additional dependencies required for using the Python inference pipelines.

```shell
pip install -e '.[pipelines]'
```

--------------------------------

### Initialize Core ML-Ready Text Decoder

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Initializes a Core ML-ready text decoder with KV cache support and optional token-level timestamp capabilities. It loads weights from a Hugging Face model and configures it for float16 precision. The example demonstrates a single decoding step.

```python
import torch
from transformers import WhisperForConditionalGeneration
from whisperkit.text_decoder import WhisperTextDecoder

hf_model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16
)
decoder = WhisperTextDecoder(hf_model.config).to(torch.float16).eval()
decoder.load_state_dict(hf_model.model.decoder.state_dict())

# Enable token-level timestamps (populates alignment head attention weights)
decoder.configure_for_token_timestamps(hf_model.generation_config)

cfg = hf_model.config
batch, n_layers, d, kv_len, enc_len = 1, cfg.decoder_layers, cfg.d_model, 448, 1500

# Single decoding step
with torch.no_grad():
    logits, key_updates, val_updates = decoder(
        input_ids=torch.tensor([50258], dtype=torch.int32),     # <|startoftranscript|>
        cache_length=torch.zeros(batch, dtype=torch.int32),
        key_cache=torch.zeros(batch, d * n_layers, 1, kv_len, dtype=torch.float16),
        value_cache=torch.zeros(batch, d * n_layers, 1, kv_len, dtype=torch.float16),
        kv_cache_update_mask=torch.cat([
            torch.ones(batch, 1), torch.zeros(batch, kv_len - 1)
        ], dim=1),
        encoder_output_embeds=torch.randn(batch, d, 1, enc_len, dtype=torch.float16),
    )[:3]
print(logits.shape)       # torch.Size([1, 1, 51865])
print(key_updates.shape)  # torch.Size([1, embed_dim*n_layers, 1, 1])
```

--------------------------------

### Shell Usage for Model Generation CLI

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Example of using the whisperkit-generate-model CLI with environment variables to specify the model repository.

```bash
# MODEL_REPO_ID=my-org/my-whisper-coreml whisperkit-generate-model \
#   --model-version openai/whisper-large-v3 --output-dir /tmp/out --upload-results
```

--------------------------------

### Install WhisperKit Android Dependencies

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Install the extra dependencies required for WhisperKit Android support. Note that this installation requires Python version less than 3.11.

```shell
pip install -e '.[android]'
```

--------------------------------

### Get Hugging Face CLI User Info

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Retrieve the currently logged-in user's information from the Hugging Face CLI. This is useful for verifying authentication before publishing models.

```shell
huggingface-cli whoami
```

--------------------------------

### Create Python Virtual Environment with Conda

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Create and activate a new Python virtual environment using Conda. This is a prerequisite for installing whisperkittools.

```shell
conda create -n whisperkit python=3.11 -y && conda activate whisperkit
```

--------------------------------

### Python Inference with WhisperKit

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Perform speech recognition inference using a unified Python wrapper for WhisperKit, whisper.cpp, mlx-examples/whisper, and WhisperOpenAIAPI. Requires installation of additional pipeline dependencies.

```python
from whisperkit.pipelines import WhisperKit, WhisperCpp, WhisperMLX, WhisperOpenAIAPI

pipe = WhisperKit(whisper_version="openai/whisper-large-v3", out_dir="/path/to/out/dir")
print(pipe("audio.{wav,flac,mp3}"))
```

--------------------------------

### get_pipeline_cls

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Factory function to dynamically select and instantiate pipeline classes based on their string names. It supports various pipeline implementations and provides examples for CLI-driven evaluation loops.

```APIDOC
## `get_pipeline_cls` — Dynamic Pipeline Selection

Factory function that resolves a string name to the appropriate pipeline class, useful for CLI-driven evaluation loops.

```python
from whisperkit.pipelines import get_pipeline_cls

for pipeline_name in ["WhisperKit", "whisper.cpp", "WhisperMLX", "WhisperOpenAIAPI"]:
    cls = get_pipeline_cls(pipeline_name)
    pipe = cls(whisper_version="openai/whisper-large-v3", out_dir=f"/tmp/{pipeline_name}")
    result = pipe("test_audio.wav")
    print(f"[{pipeline_name}] {result['text'][:60]}")

# Supported name aliases:
# "WhisperKit"                    → WhisperKit
# "whisper.cpp" / "WhisperCpp"   → WhisperCpp
# "WhisperMLX" / "mlx-whisper"   → WhisperMLX
# "WhisperHF" / "huggingface-whisper" → WhisperHF
# "WhisperHF_MPS"                → WhisperHF_MPS
# "AppleSpeechAnalyzer"          → AppleSpeechAnalyzer
# "WhisperOpenAI"                → WhisperOpenAI
# "WhisperOpenAIAPI"             → WhisperOpenAIAPI
```
```

--------------------------------

### Load and Preprocess ASR Datasets with `get_dataset`

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Shows how to use the `get_dataset` function to download and preprocess Hugging Face ASR datasets. Supports built-in, custom, and local dataset directories, handling audio path resolution, text normalization, and language filtering. The output is a list of normalized samples.

```python
from whisperkit.evaluate.datasets import get_dataset

# Load built-in debug dataset (auto-downloads from argmaxinc/librispeech-debug)
dataset = get_dataset(
    dataset_name="librispeech-debug",
    cache_dir="/tmp/datasets",
    max_num_samples=5,        # -1 for all
    language_subset=None,
)
print(len(dataset))           # 5

sample = dataset[0]
print(sample["norm_path"])    # /tmp/datasets/datasets/librispeech-debug/1234.flac
print(sample["norm_text"])    # "he hoped there would be stew for dinner..."
print(sample.get("language")) # "en"

# Load a custom HF dataset
import whisperkit._constants as c
c.DATASET_REPO_OWNER = "my-hf-org"
c.CUSTOM_EVAL_DATASET = "my-asr-testset"
dataset = get_dataset(
    dataset_name="my-asr-testset",
    cache_dir="/tmp/datasets",
)

# Load a local dataset directory (audio files + metadata.json)
c.IS_LOCAL_DATASET = True
dataset = get_dataset(
    dataset_name="/path/to/local/dataset",
    cache_dir="/tmp/datasets",
)
```

--------------------------------

### WhisperCpp Pipeline - Transcribe via whisper.cpp CLI

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Initializes the WhisperCpp pipeline, which clones and builds whisper.cpp and downloads the matching GGML model. Requires ffmpeg for non-WAV inputs.

```python
from whisperkit.pipelines import WhisperCpp

# ffmpeg must be installed: brew install ffmpeg
pipe = WhisperCpp(
    whisper_version="openai/whisper-large-v3",
    out_dir="/tmp/cpp_out"
)

# Transcribe (non-WAV files are auto-resampled to 16kHz WAV via ffmpeg)
result = pipe("interview.mp3")
print(result["text"])
# Expected: {"text": " Hello, welcome to..."}

# Quantized variant (q5_0, q8_0, q5_1, q8_1 suffixes supported)
pipe_q = WhisperCpp(
    whisper_version="openai/whisper-large-v3-q5_0",
    out_dir="/tmp/cpp_q_out"
)
result = pipe_q("audio.wav", forced_language="de")
print(result["text"])
```

--------------------------------

### Initialize Core ML-Ready Audio Encoder

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Creates a Core ML-optimized audio encoder by loading weights from a Hugging Face Whisper model and configuring it for float16 precision. It also sets up the mel spectrogram frontend.

```python
import torch
from transformers import WhisperForConditionalGeneration
from whisperkit.audio_encoder import WhisperAudioEncoder, WhisperMelSpectrogram

# Load weights from Hugging Face
hf_model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16
)
encoder = WhisperAudioEncoder(hf_model.config).to(torch.float16).eval()
encoder.load_state_dict(hf_model.model.encoder.state_dict())

# Build mel spectrogram frontend
mel = WhisperMelSpectrogram(n_mels=128, n_fft=400, hop_length=160)

# Run on random 30-second audio (480000 samples @ 16kHz)
audio = torch.randn(480000)
with torch.no_grad():
    mel_features = mel(audio)          # (1, 128, 1, 3000)
    embeddings = encoder(mel_features) # (1, 1280, 1, 1500)
print(embeddings.shape)  # torch.Size([1, 1280, 1, 1500])
```

--------------------------------

### WhisperKit Pipeline - Transcribe via Swift CLI

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Initializes the WhisperKit pipeline, which clones the repository, builds the CLI, and downloads model weights. Supports single-file and folder-batch transcription with VAD-based chunking.

```python
from whisperkit.pipelines import WhisperKit

# Initialize (clones WhisperKit repo, builds CLI, downloads model weights)
pipe = WhisperKit(
    whisper_version="openai/whisper-large-v3",
    out_dir="/tmp/wk_out",
    code_commit_hash=None,   # None → main branch
    model_commit_hash=None,  # None → latest
)

# Transcribe a single file
result = pipe("audio/interview.mp3")
print(result["text"])
# Expected: {"text": "Hello and welcome to...", "timings": {"totalDecodingFallbacks": 0}, ...}

# Transcribe with forced language and prompt
result = pipe(
    "audio/french_podcast.flac",
    forced_language="fr",
    prompt="Podcast sur la technologie:"
)
print(result["text"])

# Batch-transcribe an entire folder (single CLI invocation, more efficient)
folder_results = pipe.transcribe_folder(
    audio_folder_path="/data/recordings/",
    forced_language="en"
)
for path, res in folder_results.items():
    print(f"{path}: {res['text'][:80]}")

# Use a custom-published model
import whisperkit._constants as c
c.MODEL_REPO_ID = "my-org/my-whisper-coreml"
custom_pipe = WhisperKit(
    whisper_version="openai/whisper-large-v3",
    out_dir="/tmp/wk_custom"
)
result = custom_pipe("audio.wav")
```

--------------------------------

### Initialize Whisper HF MPS Pipeline

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Initializes the WhisperHF_MPS pipeline for processing audio with float16 precision on Apple MPS devices. Requires specifying the Whisper model version and an output directory.

```python
pipe_mps = WhisperHF_MPS(
    whisper_version="openai/whisper-large-v3",
    out_dir="/tmp/hf_mps"
)
result = pipe_mps("audio.mp3")
```

--------------------------------

### get_dataset

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Download and preprocess Hugging Face ASR datasets, handling audio paths, text normalization, and language filtering.

```APIDOC
## `get_dataset` — Dataset Loading and Preprocessing

Downloads and preprocesses a Hugging Face ASR dataset into a normalized list of samples. Handles audio path resolution, text normalization, language filtering, and ZIP extraction.

```python
from whisperkit.evaluate.datasets import get_dataset

# Load built-in debug dataset (auto-downloads from argmaxinc/librispeech-debug)
dataset = get_dataset(
    dataset_name="librispeech-debug",
    cache_dir="/tmp/datasets",
    max_num_samples=5,        # -1 for all
    language_subset=None,
)
print(len(dataset))           # 5

sample = dataset[0]
print(sample["norm_path"])    # /tmp/datasets/datasets/librispeech-debug/1234.flac
print(sample["norm_text"])    # "he hoped there would be stew for dinner..."
print(sample.get("language")) # "en"

# Load a custom HF dataset
import whisperkit._constants as c
c.DATASET_REPO_OWNER = "my-hf-org"
c.CUSTOM_EVAL_DATASET = "my-asr-testset"
dataset = get_dataset(
    dataset_name="my-asr-testset",
    cache_dir="/tmp/datasets",
)

# Load a local dataset directory (audio files + metadata.json)
c.IS_LOCAL_DATASET = True
dataset = get_dataset(
    dataset_name="/path/to/local/dataset",
    cache_dir="/tmp/datasets",
)
```
```

--------------------------------

### Lookup Prefill Cache for English Transcription

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Demonstrates how to retrieve and use prefill caches for English transcription during inference. Shows conversion between task/language IDs and flat indices.

```python
lang_token_id = prefill.tokenizer.vocab["<|en|>"]  # 50259
key_cache, val_cache = prefill(
    task=torch.tensor([0]),                        # 0=transcribe, 1=translate
    language=torch.tensor([lang_token_id])
)
print(key_cache.shape)  # torch.Size([1, embed_dim*n_layers, 1, 3])

task_idx = prefill.task_and_language_to_task_idx(task=0, language=50259)
task, lang = prefill.task_idx_to_task_and_language(task_idx)
print(f"task={task}, language_token={lang}")  # task=0, language_token=50259
```

--------------------------------

### WhisperMLX Pipeline - Apple Silicon MLX Inference

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Runs Whisper via MLX models for fast native Apple Silicon Python inference. Supported versions automatically map to mlx-community Hugging Face repositories.

```python
from whisperkit.pipelines import WhisperMLX

# Supported versions map automatically to mlx-community HF repos
pipe = WhisperMLX(
    whisper_version="openai/whisper-large-v3",
    out_dir="/tmp/mlx_out"
)

result = pipe("podcast.flac")
print(result["text"])

# With language forcing
result = pipe("audio_es.wav", forced_language="es")
print(result["text"])

# Available version → HF repo mappings:
# "openai/whisper-tiny"           → "mlx-community/whisper-tiny-mlx"
# "openai/whisper-large-v3"       → "mlx-community/whisper-large-v3-mlx"
# "openai/whisper-large-v3-turbo" → "mlx-community/whisper-large-v3-turbo"
```

--------------------------------

### Configure WhisperKit Runtime with Environment Variables

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Shows how to override default constants in whisperkit._constants using environment variables for model repositories, evaluation results, and dataset owners. Also demonstrates programmatic overrides for local model usage and API compression rates.

```python
import os

# Override the Hugging Face model repo to publish/download from
os.environ["MODEL_REPO_ID"] = "my-org/my-whisper-coreml"

# Override the evaluation results repo
os.environ["EVALS_REPO_ID"] = "my-org/my-eval-results"

# Override the dataset owner for custom HF datasets
os.environ["DATASET_REPO_OWNER"] = "my-hf-org"

# Register a custom evaluation dataset (appended to EVAL_DATASETS list)
os.environ["EVAL_DATASET"] = "my-custom-testset"
```

```python
import whisperkit._constants as c
c.IS_LOCAL_MODEL = True  # set programmatically; no env var equivalent

# Adjust OpenAI API upload compression (default: "12k")
c.OPENAI_API_COMPRESSED_UPLOAD_BIT_RATE = "8k"
```

--------------------------------

### Generate Mel Spectrograms with WhisperKit

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Demonstrates generating Mel spectrograms from audio using both standard and decomposed STFT variants.

```python
audio = torch.randn(480000)
with torch.no_grad():
    features = mel(audio)
print(features.shape)
```

```python
from whisperkit.wavlm import WhisperMelSpectrogram as DecomposedMelSpectrogram

mel_decomposed = DecomposedMelSpectrogram(n_mels=80, n_fft=400, hop_length=160)
features = mel_decomposed(audio)
print(features.shape)
```

--------------------------------

### Pre-compute KV Cache for Context Prefill

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Initializes `WhisperTextDecoderContextPrefill` to pre-compute and store decoder KV caches for various language and task combinations. This allows for instant context initialization during inference by avoiding re-running the decoder for initial tokens.

```python
import torch
from transformers import WhisperForConditionalGeneration
from whisperkit.text_decoder import WhisperTextDecoder, WhisperTextDecoderContextPrefill

hf_model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16
)
decoder = WhisperTextDecoder(hf_model.config).to(torch.float16).eval()
decoder.load_state_dict(hf_model.model.decoder.state_dict())

cfg = hf_model.config
encoder_embeds = torch.randn(4, cfg.d_model, 1, 1500, dtype=torch.float16)

# Build LUT — precomputes caches for ~200 (language, task) combos
prefill = WhisperTextDecoderContextPrefill(decoder, encoder_output_embeds=encoder_embeds)
print(f"Precomputed {len(prefill.valid_task_specs)} task specs")
```

--------------------------------

### Convert Whisper to Core ML

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Use whisperkit-generate-model to convert Hugging Face Whisper checkpoints to WhisperKit Core ML bundles. Options include quantization, context prefill, and uploading to Hugging Face Hub.

```bash
whisperkit-generate-model \
  --model-version openai/whisper-large-v3 \
  --output-dir /tmp/wk_models
```

```bash
MODEL_REPO_ID=my-org/my-whisper-repo whisperkit-generate-model \
  --model-version openai/whisper-large-v3 \
  --output-dir /tmp/wk_models \
  --generate-quantized-variants \
  --allowed-nbits 4 \
  --allowed-nbits 8 \
  --generate-decoder-context-prefill-data \
  --upload-results
```

```bash
whisperkit-generate-model \
  --model-version distil-whisper/distil-small.en \
  --output-dir /tmp/wk_models \
  --audio-encoder-sdpa-implementation SplitHeadsQ \
  --text-decoder-sdpa-implementation Cat \
  --text-decoder-max-sequence-length 224
```

```bash
MODEL_REPO_ID=my-org/my-whisper-repo whisperkit-generate-model \
  --model-version distil-whisper/distil-small.en \
  --output-dir /tmp/wk_models \
  --upload-results
```

--------------------------------

### Generate WhisperKit Core ML Model

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Convert Hugging Face PyTorch Whisper models to WhisperKit's Core ML format. Use the --model-version and --output-dir arguments. For advanced options, refer to the help menu with -h.

```shell
whisperkit-generate-model --model-version <model-version> --output-dir <output-dir>
```

--------------------------------

### Programmatic WER Evaluation with WhisperKit

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Demonstrates the use of the `evaluate` function for programmatic Word Error Rate (WER) evaluation. Shows evaluation against built-in datasets and configuration for multi-process execution. The results dictionary contains detailed metrics per file.

```python
from whisperkit.evaluate.evaluate import evaluate
from whisperkit.pipelines import WhisperKit

pipe = WhisperKit(
    whisper_version="openai/whisper-large-v3",
    out_dir="/tmp/wk_eval"
)

# Run evaluation against librispeech-debug (quick, ~10 samples)
results = evaluate(
    whisper_pipeline=pipe,
    dataset_name="librispeech-debug",
    num_samples=-1,             # -1 = all samples
    cache_dir="/tmp/datasets",
    num_proc=1,
    language_subset=None,       # None = all languages
)

# Each result dict contains:
for r in results[:2]:
    print({
        "file": r["file"],
        "reference": r["reference"],
        "prediction": r["prediction"],
        "wer": r["wer"],
        "substitution_rate": r["substitution_rate"],
        "deletion_rate": r["deletion_rate"],
        "insertion_rate": r["insertion_rate"],
        "audio_duration": r["audio_duration"],
        "prediction_duration": r["prediction_duration"],
        "num_fallbacks": r["num_fallbacks"],  # WhisperKit only
    })

# Multi-process evaluation for larger datasets
results_mp = evaluate(
    whisper_pipeline=pipe,
    dataset_name="librispeech",
    num_samples=-1,
    cache_dir="/tmp/datasets",
    num_proc=4,
)
```

--------------------------------

### Configure Qualcomm AI Hub CLI

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Configure the Qualcomm AI Hub CLI with your API token. This is required for generating models for the Android platform.

```shell
qai-hub configure --api_token
```

--------------------------------

### WhisperTextDecoderContextPrefill

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

This module pre-computes decoder KV caches for various language and task prefix combinations, storing them in embedding lookup tables for instant context initialization.

```APIDOC
## `WhisperTextDecoderContextPrefill` — Pre-computed KV Cache Lookup

Pre-computes decoder KV caches for all (language × task) token prefix combinations and stores them in `nn.Embedding` lookup tables, enabling instant context initialization at inference time without re-running the decoder for the first three tokens.

```python
import torch
from transformers import WhisperForConditionalGeneration
from whisperkit.text_decoder import WhisperTextDecoder, WhisperTextDecoderContextPrefill

hf_model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16
)
decoder = WhisperTextDecoder(hf_model.config).to(torch.float16).eval()
decoder.load_state_dict(hf_model.model.decoder.state_dict())

cfg = hf_model.config
encoder_embeds = torch.randn(4, cfg.d_model, 1, 1500, dtype=torch.float16)

# Build LUT — precomputes caches for ~200 (language, task) combos
prefill = WhisperTextDecoderContextPrefill(decoder, encoder_output_embeds=encoder_embeds)
print(f"Precomputed {len(prefill.valid_task_specs)} task specs")
```
```

--------------------------------

### WhisperOpenAIAPI Pipeline - OpenAI Cloud Transcription

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Sends audio to the OpenAI Whisper-1 API with automatic size-based compression. Results are cached to disk to avoid re-billing. Requires OPENAI_API_KEY environment variable.

```python
import os
from whisperkit.pipelines import WhisperOpenAIAPI

os.environ["OPENAI_API_KEY"] = "sk-..."

pipe = WhisperOpenAIAPI(out_dir="/tmp/openai_results")

# Transcribe (cached after first call; re-running skips API)
result = pipe("long_meeting.mp3")
# Returns verbose_json with word- and segment-level timestamps
print(result["text"])
print(result["words"][0])  # {"word": "Hello", "start": 0.0, "end": 0.32}

# Large file (>25 MB) — auto-compressed to Opus before upload
result = pipe("two_hour_lecture.wav")
print(result["text"])

# Override compression bitrate for very long files
import whisperkit._constants as c
c.OPENAI_API_COMPRESSED_UPLOAD_BIT_RATE = "8k"
result = pipe("very_long_audio.wav")
```

--------------------------------

### Whisper Mel Spectrogram Frontend

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Instantiates the `WhisperMelSpectrogram` module, a PyTorch mel spectrogram generator compatible with standard `torch.stft` and Core ML-friendly decomposed STFT paths.

```python
import torch
from whisperkit.audio_encoder import WhisperMelSpectrogram  # standard (uses torch.stft)

mel = WhisperMelSpectrogram(n_mels=128, n_fft=400, hop_length=160)
```

--------------------------------

### Generate WhisperKit Android Models

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Execute the test script to generate WhisperKit models for Android. Specify a persistent cache directory for the output.

```shell
python tests/test_aihub.py --persistent-cache-dir <output-path>
```

--------------------------------

### Publish Custom Whisper Model to Hugging Face

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Publish a custom Whisper model to Hugging Face Hub. Set the MODEL_REPO_ID environment variable to your target repository and specify the source PyTorch Whisper model. This requires write access to the Hugging Face repository.

```shell
MODEL_REPO_ID=my-org/my-whisper-repo-name whisperkit-generate-model --model-version distil-whisper/distil-small.en --output-dir <output-dir>
```

--------------------------------

### Evaluate WhisperKit Model on Custom Dataset

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Evaluate WhisperKit models on your own dataset published on the Hugging Face Hub. Ensure your dataset has audio files and a `metadata.json`.

```shell
export CUSTOM_EVAL_DATASET="my-dataset-name-on-hub"
export DATASET_REPO_OWNER="my-user-or-org-name-on-hub"
export MODEL_REPO_ID="my-org/my-whisper-repo-name" # if evaluating self-published models
whisperkit-evaluate-model --model-version <model-version> --output-dir <output-dir> --evaluation-dataset my-dataset-name-on-hub
```

--------------------------------

### Evaluate WhisperKit Model

Source: https://github.com/argmaxinc/whisperkittools/blob/main/README.md

Use this command to evaluate WhisperKit models on specified datasets. It defaults to the latest `main` branch commits and searches Argmax-published model repositories.

```shell
whisperkit-evaluate-model --model-version <model-version> --output-dir <output-dir> --evaluation-dataset {librispeech-debug,librispeech,earnings22}
```

--------------------------------

### Quantize Whisper Audio Encoder and Text Decoder

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Utilizes WhisperAudioEncoderPalettizer and WhisperTextDecoderPalettizer for palettization (lookup-table quantization) of Whisper sub-models. Quality is tracked via PSNR for the encoder and KL divergence for the decoder. The `run()` method generates artifacts, and `plot_specs` visualizes response curves.

```python
from whisperkit.compress.palettize import (
    WhisperAudioEncoderPalettizer,
    WhisperTextDecoderPalettizer,
)

# Palettize AudioEncoder — quality measured by PSNR
ae_palettizer = WhisperAudioEncoderPalettizer(
    model_version="openai/whisper-large-v3",
    allowed_nbits=[4, 8],
    group_size=32,            # per-group palettization
    outlier_decomp=False,
)
ae_palettizer.run()           # generates .mlmodelc artifacts in output dir

# Palettize TextDecoder — quality measured by KL divergence on logits
td_palettizer = WhisperTextDecoderPalettizer(
    model_version="openai/whisper-large-v3",
    allowed_nbits=[4, 8],
    group_size=32,
    outlier_decomp=True,      # sparse outlier decomposition for better quality
)
td_palettizer.run()

# Plot response curves (model size reduction % vs output divergence)
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
td_palettizer.plot_specs(fig, ax)
plt.savefig("td_response_curves.png")
```

--------------------------------

### Evaluate Whisper Model WER

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Run whisperkit-evaluate-model for WER benchmarking against datasets. Supports various pipelines, parallel processing, and uploading results. Custom datasets and models can also be evaluated.

```bash
whisperkit-evaluate-model \
  --model-version openai/whisper-large-v3 \
  --output-dir /tmp/evals \
  --evaluation-dataset librispeech-debug \
  --pipeline WhisperKit
```

```bash
whisperkit-evaluate-model \
  --model-version openai/whisper-large-v3 \
  --output-dir /tmp/evals \
  --evaluation-dataset librispeech \
  --pipeline WhisperKit \
  --num-proc 4 \
  --upload-results
```

```bash
export CUSTOM_EVAL_DATASET="my-asr-testset"
export DATASET_REPO_OWNER="my-hf-org"
export MODEL_REPO_ID="my-hf-org/my-whisper-coreml"
whisperkit-evaluate-model \
  --model-version openai/whisper-large-v3 \
  --output-dir /tmp/evals \
  --evaluation-dataset my-asr-testset \
  --pipeline WhisperKit
```

```bash
whisperkit-evaluate-model \
  --model-version openai/whisper-large-v3 \
  --output-dir /tmp/evals \
  --evaluation-dataset earnings22-debug \
  --pipeline whisper.cpp
```

--------------------------------

### WhisperHF / WhisperHF_MPS Pipelines - Hugging Face Transformers

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Runs Whisper via transformers.pipeline on CUDA (auto-detected) or Apple MPS. Results include text and chunk-level timestamps.

```python
from whisperkit.pipelines import WhisperHF, WhisperHF_MPS
import torch

# CUDA (auto) or CPU
pipe = WhisperHF(
    whisper_version="openai/whisper-large-v3",
    out_dir="/tmp/hf_out"
)
result = pipe("audio.wav")
print(result["text"])
print(result["chunks"])  # [{"timestamp": [0.0, 2.5], "text": " Hello"}, ...]
```

--------------------------------

### WhisperAudioEncoderPalettizer and WhisperTextDecoderPalettizer

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Quantize Whisper sub-models using palettization for reduced model size while monitoring output quality via PSNR or KL divergence.

```APIDOC
## WhisperAudioEncoderPalettizer / WhisperTextDecoderPalettizer — Quantization

Subclasses of `argmaxtools.compress.palettize.Palettizer` that perform palettization (lookup-table quantization) of Whisper sub-models while tracking output quality via PSNR (encoder) or KL divergence (decoder).

```python
from whisperkit.compress.palettize import (
    WhisperAudioEncoderPalettizer,
    WhisperTextDecoderPalettizer,
)

# Palettize AudioEncoder — quality measured by PSNR
ae_palettizer = WhisperAudioEncoderPalettizer(
    model_version="openai/whisper-large-v3",
    allowed_nbits=[4, 8],
    group_size=32,            # per-group palettization
    outlier_decomp=False,
)
ae_palettizer.run()           # generates .mlmodelc artifacts in output dir

# Palettize TextDecoder — quality measured by KL divergence on logits
td_palettizer = WhisperTextDecoderPalettizer(
    model_version="openai/whisper-large-v3",
    allowed_nbits=[4, 8],
    group_size=32,
    outlier_decomp=True,      # sparse outlier decomposition for better quality
)
td_palettizer.run()

# Plot response curves (model size reduction % vs output divergence)
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
td_palettizer.plot_specs(fig, ax)
plt.savefig("td_response_curves.png")
```
```

--------------------------------

### Dynamically Select Pipeline Class

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Uses `get_pipeline_cls` to dynamically select and instantiate different Whisper pipeline classes based on their string names. This is useful for creating flexible evaluation loops, especially in CLI applications. Supported aliases are provided for convenience.

```python
from whisperkit.pipelines import get_pipeline_cls

for pipeline_name in ["WhisperKit", "whisper.cpp", "WhisperMLX", "WhisperOpenAIAPI"]:
    cls = get_pipeline_cls(pipeline_name)
    pipe = cls(whisper_version="openai/whisper-large-v3", out_dir=f"/tmp/{pipeline_name}")
    result = pipe("test_audio.wav")
    print(f"[{pipeline_name}] {result['text'][:60]}")
```

--------------------------------

### WhisperMelSpectrogram

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

PyTorch module for generating Mel spectrograms, compatible with standard PyTorch and Core ML workflows.

```APIDOC
## `WhisperMelSpectrogram` — Mel Spectrogram Frontend

PyTorch mel spectrogram module compatible with both the standard `torch.stft` path and a Core ML-friendly decomposed STFT path (used for Android via `whisperkit.wavlm`).

```python
import torch
from whisperkit.audio_encoder import WhisperMelSpectrogram  # standard (uses torch.stft)

mel = WhisperMelSpectrogram(n_mels=128, n_fft=400, hop_length=160)
```
```

--------------------------------

### WhisperAudioEncoder

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

A PyTorch nn.Module that implements Whisper's audio encoder, optimized for Core ML tracing. It ingests mel-spectrogram tensors and outputs encoder embeddings.

```APIDOC
## `WhisperAudioEncoder` — Core ML-Ready Audio Encoder

PyTorch `nn.Module` that reproduces Whisper's audio encoder with Conv2d-mapped weights optimized for Core ML tracing. Ingests 4D mel-spectrogram tensors `(batch, num_mel_bins, 1, 2*encoder_seq_len)` and outputs encoder embeddings `(batch, embed_dim, 1, encoder_seq_len)`.

```python
import torch
from transformers import WhisperForConditionalGeneration
from whisperkit.audio_encoder import WhisperAudioEncoder, WhisperMelSpectrogram

# Load weights from Hugging Face
hf_model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16
)
encoder = WhisperAudioEncoder(hf_model.config).to(torch.float16).eval()
encoder.load_state_dict(hf_model.model.encoder.state_dict())

# Build mel spectrogram frontend
mel = WhisperMelSpectrogram(n_mels=128, n_fft=400, hop_length=160)

# Run on random 30-second audio (480000 samples @ 16kHz)
audio = torch.randn(480000)
with torch.no_grad():
    mel_features = mel(audio)          # (1, 128, 1, 3000)
    embeddings = encoder(mel_features) # (1, 1280, 1, 1500)
print(embeddings.shape)  # torch.Size([1, 1280, 1, 1500])
```
```

--------------------------------

### evaluate

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

Programmatically evaluate WhisperKit models by calculating Word Error Rate (WER) against specified datasets.

```APIDOC
## `evaluate` — Programmatic WER Evaluation

Core evaluation function used by the `whisperkit-evaluate-model` CLI. Can be called directly to run custom evaluation loops.

```python
from whisperkit.evaluate.evaluate import evaluate
from whisperkit.pipelines import WhisperKit

pipe = WhisperKit(
    whisper_version="openai/whisper-large-v3",
    out_dir="/tmp/wk_eval"
)

# Run evaluation against librispeech-debug (quick, ~10 samples)
results = evaluate(
    whisper_pipeline=pipe,
    dataset_name="librispeech-debug",
    num_samples=-1,             # -1 = all samples
    cache_dir="/tmp/datasets",
    num_proc=1,
    language_subset=None,       # None = all languages
)

# Each result dict contains:
for r in results[:2]:
    print({
        "file": r["file"],
        "reference": r["reference"],
        "prediction": r["prediction"],
        "wer": r["wer"],
        "substitution_rate": r["substitution_rate"],
        "deletion_rate": r["deletion_rate"],
        "insertion_rate": r["insertion_rate"],
        "audio_duration": r["audio_duration"],
        "prediction_duration": r["prediction_duration"],
        "num_fallbacks": r["num_fallbacks"],  # WhisperKit only
    })

# Multi-process evaluation for larger datasets
results_mp = evaluate(
    whisper_pipeline=pipe,
    dataset_name="librispeech",
    num_samples=-1,
    cache_dir="/tmp/datasets",
    num_proc=4,
)
```
```

--------------------------------

### WhisperTextDecoder

Source: https://context7.com/argmaxinc/whisperkittools/llms.txt

An autoregressive text decoder module that supports explicit KV cache inputs/outputs and optional token-level timestamp generation through alignment heads.

```APIDOC
## `WhisperTextDecoder` — Core ML-Ready Text Decoder with KV Cache

Autoregressive decoder with explicit KV cache inputs/outputs and optional token-level timestamp support via alignment heads.

```python
import torch
from transformers import WhisperForConditionalGeneration
from whisperkit.text_decoder import WhisperTextDecoder

hf_model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16
)
decoder = WhisperTextDecoder(hf_model.config).to(torch.float16).eval()
decoder.load_state_dict(hf_model.model.decoder.state_dict())

# Enable token-level timestamps (populates alignment head attention weights)
decoder.configure_for_token_timestamps(hf_model.generation_config)

cfg = hf_model.config
batch, n_layers, d, kv_len, enc_len = 1, cfg.decoder_layers, cfg.d_model, 448, 1500

# Single decoding step
with torch.no_grad():
    logits, key_updates, val_updates = decoder(
        input_ids=torch.tensor([50258], dtype=torch.int32),     # <|startoftranscript|>
        cache_length=torch.zeros(batch, dtype=torch.int32),
        key_cache=torch.zeros(batch, d * n_layers, 1, kv_len, dtype=torch.float16),
        value_cache=torch.zeros(batch, d * n_layers, 1, kv_len, dtype=torch.float16),
        kv_cache_update_mask=torch.cat([
            torch.ones(batch, 1), torch.zeros(batch, kv_len - 1)
        ], dim=1),
        encoder_output_embeds=torch.randn(batch, d, 1, enc_len, dtype=torch.float16),
    )[:3]
print(logits.shape)       # torch.Size([1, 1, 51865])
print(key_updates.shape)  # torch.Size([1, embed_dim*n_layers, 1, 1])
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.