llamafile

https://docs.mozilla.ai/llamafile
Admin
llamafile lets you distribute and run LLMs with a single file. It combines llama.cpp with...
Tokens:14,897
Snippets:201
Trust Score:9.5
Update:4 months ago
Show doc for...
Tokens:
Raw
### Setup Llamafile Repository

Source: https://docs.mozilla.ai/llamafile/using-llamafile/building_dlls.md

Navigates into the cloned llamafile directory and runs the 'make setup' command to initialize submodules and apply patches.

```bash
cd llamafile
make setup
```

--------------------------------

### Install Whisperfile Systemwide

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Install the whisperfile software as a systemwide command using the make install command after building.

```bash
.cosmocc/4.0.2/bin/make -j8
sudo make install
```

--------------------------------

### Python Client Example

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

This example demonstrates how to use the `openai` Python package to interact with the Llamafile API. By changing the `base_url` and `api_key`, you can direct your existing OpenAI-compatible applications to use Llamafile.

```APIDOC
## Python Client

### Description
Utilizes the `openai` Python package to communicate with the Llamafile API by configuring the `base_url` and `api_key`.

### Usage
Ensure you have installed the package: `pip3 install openai`.

### Code Example
```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
    api_key = "sk-no-key-required"
)

completion = client.chat.completions.create(
    model="LLaMA_CPP",
    messages=[
        {"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with your requests."},
        {"role": "user", "content": "Write a limerick about python exceptions"}
    ]
)

print(completion.choices[0].message)
```

### Response Example
```python
ChatCompletionMessage(content='A script that crashes like a ghost,\nWhen it tries to solve the problem deep and fast.\nThe error message pops up in a bright light,\nAnd tells us what\'s wrong when we try to fix it.', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None)
```
```

--------------------------------

### Clone Repository and Setup Submodules

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Clone the llamafile repository and initialize all submodules, which is a required step before building.

```bash
git clone https://github.com/mozilla-ai/llamafile.git
cd llamafile

# initialise all submodules - this step is required,
as the submodules need to be pulled and patched first!
make setup
```

--------------------------------

### Server Health Status (Ready)

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Example JSON response indicating the server is ready.

```json
{"status": "ok"}
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Perform an HTTP GET request on a documentation page URL with the 'ask' query parameter to dynamically query the documentation.

```http
GET https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md?ask=<question>
```

--------------------------------

### Download Model Weights

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Download the desired model weights. This example uses the tiny q5_1 quantized weights from Hugging Face.

```bash
curl -LO https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en-q5_1.bin
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.mozilla.ai/llamafile/reference/support.md

Send an HTTP GET request to the current page URL with the 'ask' query parameter to ask a question about the documentation. The question should be specific and self-contained.

```http
GET https://docs.mozilla.ai/llamafile/reference/support.md?ask=<question>
```

--------------------------------

### Install MSYS2 Tools

Source: https://docs.mozilla.ai/llamafile/using-llamafile/building_dlls.md

Installs essential tools like git, patch, unzip, wget, make, and vim within the MSYS shell environment for setting up the Llamafile repository.

```bash
pacman -S git patch unzip wget make vim
```

--------------------------------

### Build and Install Llamafile

Source: https://docs.mozilla.ai/llamafile/using-llamafile/source_installation.md

Compile Llamafile using the downloaded Cosmopolitan C compiler and install it to a specified prefix. The `-j8` flag enables parallel compilation.

```shell
.cosmocc/4.0.2/bin/make -j8
sudo .cosmocc/4.0.2/bin/make install PREFIX=/usr/local
```

--------------------------------

### Download MP3 Audio File

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Download an example MP3 audio file for testing transcription with different formats.

```bash
curl -LO https://archive.org/download/raven/raven_poe_64kb.mp3
```

--------------------------------

### Install APE loader on WSL

Source: https://docs.mozilla.ai/llamafile/reference/troubleshooting.md

Ensures the APE loader is installed to `/usr/bin/ape` and is executable. This is a prerequisite for the systemd service configuration.

```shell
sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo chmod +x /usr/bin/ape
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

Perform an HTTP GET request to query the documentation dynamically. Include your question as the 'ask' query parameter.

```bash
GET https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md?ask=<question>
```

--------------------------------

### Python API Client Example

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

This Python script uses the `openai` package to interact with the Llamafile API. Make sure to install the package using `pip3 install openai` and configure the `base_url` and `api_key`.

```python
#!/usr/bin/env python3
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
    api_key = "sk-no-key-required"
)
completion = client.chat.completions.create(
    model="LLaMA_CPP",
    messages=[
        {"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
        {"role": "user", "content": "Write a limerick about python exceptions"}
    ]
)
print(completion.choices[0].message)
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Perform an HTTP GET request to the documentation URL with the `ask` query parameter to dynamically query the documentation.

```http
GET https://docs.mozilla.ai/llamafile/whisperfile/packaging.md?ask=<question>
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.mozilla.ai/llamafile/using-llamafile

Perform an HTTP GET request to query the documentation with a specific question. The response includes a direct answer and relevant excerpts.

```http
GET https://docs.mozilla.ai/llamafile/using-llamafile.md?ask=<question>
```

--------------------------------

### Starting Whisperfile HTTP Server

Source: https://docs.mozilla.ai/llamafile/whisperfile/index.md

Starts the whisper-server with a specified model and port. This allows for remote transcription via a REST API.

```sh
whisper-server -m whisper-tiny.en-q5_1.bin --port 8080
```

--------------------------------

### Run text-only llamafile as server

Source: https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles.md

Starts the created text-only llamafile as a web server. This allows for API-based interaction with the model.

```shell
./Qwen3-0.6B-Q8.llamafile --server
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.mozilla.ai/llamafile/using-llamafile/source_installation.md

Perform an HTTP GET request to query the documentation with a natural language question.

```http
GET https://docs.mozilla.ai/llamafile/using-llamafile/source_installation.md?ask=<question>
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

Use this endpoint to ask questions about the documentation. The response includes direct answers and source excerpts.

```http
GET https://docs.mozilla.ai/llamafile/getting-started/quickstart.md?ask=<question>
```

--------------------------------

### Running llamafile in server mode for AI agents with Jinja templating

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

This example configures a llamafile to serve models for AI agents. It enables Jinja templating and sets a large context size for extensive processing.

```sh
./gpt-oss-20b-mxfp4.llamafile \
  --server \
  --host 0.0.0.0
  --jinja
  --ctx-size 64000
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.mozilla.ai/llamafile/reference

To get additional information not directly available on a page, perform an HTTP GET request to the page URL with the 'ask' query parameter. The question should be specific and in natural language.

```http
GET https://docs.mozilla.ai/llamafile/reference.md?ask=<question>
```

--------------------------------

### Run the Llamafile

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

Execute the downloaded llamafile from your terminal to start the local LLM chat interface. This command initiates the model and makes it available for interaction.

```sh
./Qwen3.5-0.8B-Q8_0.llamafile
```

--------------------------------

### Query Documentation with GET Request

Source: https://docs.mozilla.ai/llamafile/index.md

Use this method to ask specific questions about the documentation when the answer is not explicitly present. The question should be in natural language and self-contained.

```http
GET https://docs.mozilla.ai/llamafile/index.md?ask=<question>
```

--------------------------------

### Query documentation dynamically

Source: https://docs.mozilla.ai/llamafile/whisperfile/gpu.md

Perform an HTTP GET request to the current page URL with the `ask` query parameter to dynamically query the documentation. The question should be specific and in natural language.

```http
GET https://docs.mozilla.ai/llamafile/whisperfile/gpu.md?ask=<question>
```

--------------------------------

### Configure systemd service for APE loader on WSL

Source: https://docs.mozilla.ai/llamafile/reference/troubleshooting.md

This systemd service configuration ensures the APE loader is registered when the system starts on WSL. It should be placed in `/etc/systemd/system/cosmo-binfmt.service`.

```systemd
[Unit]
Description=cosmopolitan APE binfmt service
After=wsl-binfmt.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"

[Install]
WantedBy=multi-user.target
```

--------------------------------

### Server Health Status (Loading)

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Example JSON response indicating the server is still loading the model.

```json
{"status": "loading model"}
```

--------------------------------

### Model Load Success Response

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Example HTTP 200 response indicating successful model loading.

```text
Load was successful!
```

--------------------------------

### Transcription Response

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Example JSON response from the /inference endpoint containing the transcribed text.

```json
{"text": " And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country."}
```

--------------------------------

### Build ROCm DLL (Parallel)

Source: https://docs.mozilla.ai/llamafile/using-llamafile/building_dlls.md

Starts the parallel build process for the ROCm DLL. This command is executed from the llamafile directory.

```bash
llamafile\rocm_parallel.bat
```

--------------------------------

### Running llamafile in CLI mode for image description

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

This example shows how to use llamafile in CLI mode with a multimodal model to describe an image. It requires specifying the model weights, mmproj weights, and the image path.

```sh
llamafile -ngl 9999 --temp 0 \
  --cli
  --image ~/Pictures/lemurs.jpg \
  -m llava-v1.6-mistral-7b.Q4_K_M.gguf \
  --mmproj mmproj-model-f16.gguf \
  -p 'Describe this picture'
```

--------------------------------

### Download Tiny Whisper Model

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Download the tiny quantized whisper model weights for quick setup and fast transcription. This model offers good, though not perfect, accuracy.

```bash
curl -L -o models/whisper-tiny.en-q5_1.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en-q5_1.bin
```

--------------------------------

### Running llamafile in CLI mode for web page summarization

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

This example shows how to use llamafile in CLI mode to summarize a web page. It pipes the output of 'links' command to the prompt.

```sh
./Qwen3.5-9B-Q5_K_S.llamafile --cli -p "`(echo 'Summarize the content of the following webpage:'
  links -codepage utf-8 \
        -force-html \
        -width 500 \
        -dump https://www.poetryfoundation.org/poems/48860/the-raven | 
    sed 's/   */ /g')`"
```

--------------------------------

### Interactive Chatbot with Llamafile

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

Run Llamafile as an interactive chatbot using a specific model and prompt. This example configures interactive mode, colors, batch size, context size, temperature, and reverse prompts for a conversation.

```sh
llamafile -m llama-65b-Q5_K.gguf -p '
The following is a conversation between a Researcher and their helpful AI assistant Digital Athena which is a large language model trained on the sum of human knowledge.
Researcher: Good morning.
Digital Athena: How can I help you today?
Researcher:' --interactive --color --batch_size 1024 --ctx_size 4096 \
--keep -1 --temp 0 --mirostat 2 --in-prefix ' ' --interactive-first \
--in-suffix 'Digital Athena:' --reverse-prompt 'Researcher:'

```

--------------------------------

### Whisperfile Command-Line Usage

Source: https://docs.mozilla.ai/llamafile/whisperfile/index.md

Examples of using the whisperfile command-line tool for transcription and translation. Ensure the model file is present in the current directory or specify its path.

```sh
whisperfile -m whisper-tiny.en-q5_1.bin audio.wav
```

```sh
whisperfile -m ggml-medium-q5_0.bin -f audio.ogg --translate
```

--------------------------------

### Running llamafile in server mode with custom host and port

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

This command starts a llamafile in server mode, making it accessible over the network. It specifies the host address and a custom port.

```sh
./llava-v1.6-mistral-7b-Q4_K_M.llamafile \
  --server \
  --host 0.0.0.0 \
  --port 8081
```

--------------------------------

### Curl API Client Example

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

Use this curl command to send a chat completion request to the Llamafile API. Ensure the API server is running and accessible at http://localhost:8080.

```shell
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d "{
  "model": "LLaMA_CPP",
  "messages": [
      {
          "role": "system",
          "content": "You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
      },
      {
          "role": "user",
          "content": "Write a limerick about python exceptions"
      }
    ]
}" | python3 -c '
import json
import sys
json.dump(json.load(sys.stdin), sys.stdout, indent=2)
print()
'
```

--------------------------------

### Grant Execute Permissions (macOS/Linux/BSD)

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

Use this command in your terminal to make the downloaded llamafile executable on macOS, Linux, or BSD systems. This is a one-time setup step.

```sh
chmod +x Qwen3.5-0.8B-Q8_0.llamafile
```

--------------------------------

### Generate C Code with WizardCoder

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

Use Llamafile with WizardCoder-Python-13B weights to generate C code for a libc function like memcpy. This example specifies temperature, reverse prompts, and the initial prompt.

```sh
llamafile \
  -m wizardcoder-python-13b-v1.0.Q8_0.gguf \
  --temp 0 -r '}\n' -r '```\n' \
  -e -p '```c\nvoid *memcpy(void *dst, const void *src, size_t size) {\n'
```

--------------------------------

### Enforce Output with BNF Grammar for Image Renaming

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

Utilize Llamafile with a multimodal model (Llava) and BNF grammar to generate predictable output for renaming image files. This example includes GPU layers, temperature, image input, model and mmproj paths, grammar rules, and prompt, redirecting stderr and piping to sed for formatting.

```sh
llamafile -ngl 9999 --temp 0 \
    --image lemurs.jpg \
    -m llava-v1.5-7b-Q4_K.gguf \
    --mmproj llava-v1.5-7b-mmproj-Q4_0.gguf \
    --grammar 'root ::= [a-z]+ (" " [a-z]+)+' \
    -e -p '### User: What do you see?\n### Assistant: ' \
    --no-display-prompt 2>/dev/null |
  sed -e's/ /_/g' -e's/$/.jpg/'
a_baby_monkey_on_the_back_of_a_mother.jpg
```

--------------------------------

### GET /health

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Checks the health status of the whisper-server. It returns an HTTP 503 if the model is still loading, otherwise it returns an HTTP 200.

```APIDOC
## GET /health

### Description
Returns server health status as JSON. Returns HTTP 503 if the model is still loading.

### Method
GET

### Endpoint
/health

### Response
#### Success Response (200)
- **status** (string) - Indicates the server is ready, e.g., "ok"

#### Response Example
```json
{
  "status": "ok"
}
```

#### Loading Response (503)
- **status** (string) - Indicates the model is loading, e.g., "loading model"

#### Response Example
```json
{
  "status": "loading model"
}
```
```

--------------------------------

### Chat Completions API (v1/chat/completions)

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

This endpoint is compatible with the OpenAI API's chat completions endpoint. It allows you to send messages to a model and receive a chat completion response. The example demonstrates how to use curl to interact with this endpoint.

```APIDOC
## POST /v1/chat/completions

### Description
Sends messages to a model to receive a chat completion response, compatible with the OpenAI API.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Request Body
- **model** (string) - Required - The model to use for completion.
- **messages** (array) - Required - A list of message objects representing the conversation history.
  - **role** (string) - Required - The role of the message sender (e.g., "system", "user").
  - **content** (string) - Required - The content of the message.

### Request Example
```json
{
  "model": "LLaMA_CPP",
  "messages": [
      {
          "role": "system",
          "content": "You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
      },
      {
          "role": "user",
          "content": "Write a limerick about python exceptions"
      }
    ]
}
```

### Response
#### Success Response (200)
- **choices** (array) - An array of chat completion choices.
  - **message** (object) - The generated message.
    - **role** (string) - The role of the message sender.
    - **content** (string) - The content of the message.
- **created** (integer) - The timestamp of when the completion was created.
- **model** (string) - The model used for the completion.
- **id** (string) - The unique identifier for the completion.
- **object** (string) - The type of object returned.
- **usage** (object) - Usage statistics for the completion.
  - **completion_tokens** (integer) - Number of tokens in the completion.
  - **prompt_tokens** (integer) - Number of tokens in the prompt.
  - **total_tokens** (integer) - Total number of tokens used.

#### Response Example
```json
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "In the world of Python, where magic breaks and errors occur,\nA script fails when it should not have failed.\nWith a `KeyError`, I can\'t access the key,\nSo I tell you to use the `except` clause!"
      }
    }
  ],
  "created": 1773659260,
  "model": "Qwen3.5-0.8B-Q8_0.gguf",
  "system_fingerprint": "b1773565177-7f5ee5496",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 52,
    "prompt_tokens": 49,
    "total_tokens": 101
  },
  "id": "chatcmpl-KOqwN6C0oRzINGZuFqZ95bU1iPfc6RFO",
  "timings": {
    "cache_n": 0,
    "prompt_n": 49,
    "prompt_ms": 54.944,
    "prompt_per_token_ms": 1.1213061224489795,
    "prompt_per_second": 891.8171228887594,
    "predicted_n": 52,
    "predicted_ms": 405.856,
    "predicted_per_token_ms": 7.804923076923076,
    "predicted_per_second": 128.1242608215722
  }
}
```
```

--------------------------------

### Initialize and Configure Dependencies

Source: https://docs.mozilla.ai/llamafile/using-llamafile/source_installation.md

Run this command to set up git submodules and download the Cosmopolitan C compiler before building.

```shell
make setup
```

--------------------------------

### Create .args File for Default Arguments

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Create a `.args` file to specify default arguments, such as the model path. Each argument should be on a new line. The `...` token represents runtime arguments.

```text
-m
/zip/ggml-tiny.en-q5_1.bin
...
```

--------------------------------

### Create Build Workspace

Source: https://docs.mozilla.ai/llamafile/using-llamafile/building_dlls.md

Creates a directory for your build workspace and navigates into it. Replace 'Your_Username' with your actual Windows username.

```bash
mkdir /c/Users/Your_Username/workspace
cd /c/Users/Your_Username/workspace
```

--------------------------------

### Running a pre-bundled llamafile

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

This command shows how to execute a pre-bundled llamafile, which is equivalent to running a llamafile with specified model weights.

```sh
./Apertus-8B-Instruct-2509.llamafile --temp ...
```

--------------------------------

### Build and Run Whisper Server

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Build the whisper-server and run it with a specified model. Ensure you have the correct build environment set up.

```bash
.cosmocc/4.0.2/bin/make -j8 o//whisperfile
o//whisperfile/whisper-server -m models/whisper-tiny.en-q5_1.bin
```

--------------------------------

### Running a pre-bundled multimodal llamafile in CLI mode

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

This command demonstrates running a pre-bundled multimodal llamafile in CLI mode for image description, simplifying the process by not requiring separate mmproj weights.

```sh
./Ministral-3-3B-Instruct-2512-Q4_K_M.llamafile -ngl 9999 \
  --cli
  --image ~/Pictures/lemurs.jpg \
  -p 'Describe this picture'
```

--------------------------------

### Running llamafile with model weights

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

This command demonstrates how to run a llamafile by specifying the model weights directly. It's equivalent to using a pre-bundled llamafile.

```sh
llamafile -m Apertus-8B-Instruct-2509.gguf --temp ...
```

--------------------------------

### Create and bundle multimodal llamafile

Source: https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles.md

Copies the llamafile executable and uses zipalign to embed multimodal model weights, projector, and arguments into a new llamafile. This creates a self-contained multimodal LLM executable.

```shell
cp o//llamafile/llamafile llava.llamafile

o//third_party/zipalign/zipalign -j0 \
  llava.llamafile \
  llava-v1.6-mistral-7b.Q8_0.gguf \
  mmproj-model-f16.gguf \
  .args

./llava.llamafile
```

--------------------------------

### Create and bundle text-only llamafile

Source: https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles.md

Copies the llamafile executable and uses zipalign to embed model weights and arguments into a new llamafile. This creates a shareable, self-contained LLM executable.

```shell
cp o//llamafile/llamafile Qwen3-0.6B-Q8.llamafile

o//third_party/zipalign/zipalign -j0 \
  Qwen3-0.6B-Q8.llamafile \
  Qwen3-0.6B-Q8_0.gguf \
  .args

./Qwen3-0.6B-Q8.llamafile
```

--------------------------------

### Verify Build with Unit Tests

Source: https://docs.mozilla.ai/llamafile/using-llamafile/source_installation.md

Execute this command to run unit tests and confirm that Llamafile has been built correctly.

```shell
make check
```

--------------------------------

### Whisper Server Options

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Lists the available command-line options for configuring the whisper-server. Use --help for a complete list.

```text
whisper-server options:
  -m FNAME, --model FNAME     Path of Whisper model weights
  --host ADDR                 Hostname or IP address to bind to (default: 127.0.0.1)
  --port PORT                 Port number (default: 8080)
  -l LANG, --language LANG    Default spoken language ('auto' for auto-detect)
  -tr, --translate            Translate audio into English text
  -t N, --threads N           Number of threads to use during computation
  -ng, --no-gpu               Disable GPU acceleration
  --gpu VALUE                 Select GPU backend (auto, apple, amd, nvidia, disable)
  --log-disable               Suppress logging output
```

--------------------------------

### Build Whisperfile Executable

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Build the `whisperfile` executable from source if a prebuilt version is not available. It is then copied with a more specific name for clarity.

```bash
.cosmocc/4.0.2/bin/make -j8 o//whisperfile

# copy it with a more specific name
cp o//whisperfile/whisperfile whisper-tiny
```

--------------------------------

### Querying Documentation Dynamically

Source: https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles.md

Use this method to ask questions about the documentation when the answer is not explicitly present on the page. The response includes a direct answer and relevant excerpts.

```bash
GET https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles.md?ask=<question>
```

--------------------------------

### Build Whisperfile Software

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Compile the whisperfile software from source using the provided build command.

```bash
.cosmocc/4.0.2/bin/make -j8 o//whisperfile
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.mozilla.ai/llamafile/getting-started/pre-built-llamafiles.md

Use this method to ask questions about the documentation when the answer is not explicitly present on the current page. The question should be specific and in natural language.

```http
GET https://docs.mozilla.ai/llamafile/getting-started/pre-built-llamafiles.md?ask=<question>
```

--------------------------------

### Run Whisperfile with Embedded Weights

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Execute the bundled whisperfile, specifying the embedded model weights using the `/zip/` path prefix. A sample audio file is used for transcription.

```bash
./whisper-tiny -m /zip/ggml-tiny.en-q5_1.bin -f whisper.cpp/samples/jfk.wav
```

--------------------------------

### Download llamafile and External Weights (Windows)

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

Download the llamafile executable and a large GGUF model for use on Windows, bypassing executable file size limits. Ensure you use the correct path separator (\ for Windows).

```shell
curl -L -o llamafile.exe https://huggingface.co/mozilla-ai/llamafile_0.10/resolve/main/llamafile_0.10.1
curl -L -o gpt-oss.gguf https://huggingface.co/unsloth/gpt-oss-20b-GGUF/resolve/main/gpt-oss-20b-Q5_K_S.gguf
./llamafile.exe -m gpt-oss.gguf
```

--------------------------------

### Embed .args File and Create Self-Contained Executable

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Embed the `.args` file into the whisperfile using `zipalign` to create a self-contained executable. The `.args` file is then removed.

```bash
o//third_party/zipalign/zipalign whisper-tiny .args
rm -f .args
```

--------------------------------

### Multimodal llamafile arguments

Source: https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles.md

Defines arguments for a multimodal llamafile, including paths to the main model, a multimodal projector, and server-related configurations. The '/zip/' prefix is required for files packaged within the llamafile.

```shell
-m
/zip/llava-v1.6-mistral-7b.Q8_0.gguf
--mmproj
/zip/mmproj-model-f16.gguf
--server
--host
0.0.0.0
-ngl
9999
--no-mmap
...
```

--------------------------------

### Run Llamafile CLI with Prompt

Source: https://docs.mozilla.ai/llamafile/using-llamafile/source_installation.md

Execute the compiled Llamafile in CLI mode, providing a prompt and a GGUF model.

```shell
./o/llamafile/llamafile --model <gguf_model> --cli -p "Hello world"
```

--------------------------------

### Load Model at Runtime

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Use the /load endpoint to dynamically load a different Whisper model while the server is running. Provide the path to the new model weights.

```bash
curl http://localhost:8080/load \
  -F "model=/path/to/model.bin"
```

--------------------------------

### Build zipalign Tool

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Build the `zipalign` tool, which is necessary for embedding files into the executable without corrupting its ZIP structure.

```bash
.cosmocc/4.0.2/bin/make -j8 o//third_party/zipalign
```

--------------------------------

### Run llamafile with LM Studio Models

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

Execute llamafile using a model downloaded and managed by LM Studio. The path points to the GGUF file within LM Studio's cache directory.

```shell
llamafile -m ~/.cache/lm-studio/models/lmstudio-community/gpt-oss-20b-GGUF/gpt-oss-20b-MXFP4.gguf
```

--------------------------------

### Run Llamafile as a Server

Source: https://docs.mozilla.ai/llamafile/using-llamafile/source_installation.md

Run the compiled Llamafile executable in server mode with a specified GGUF model.

```shell
./o/llamafile/llamafile --model <gguf_model> --server
```

--------------------------------

### Download Multilingual Model and Audio

Source: https://docs.mozilla.ai/llamafile/whisperfile/translate.md

Download a suitable multilingual Whisperfile model and an audio file for testing translation. Ensure the model does not have '.en' in its name for translation capabilities.

```bash
curl -LO https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium-q5_0.bin
curl -LO https://archive.org/download/avventure_pinocchio_librivox/avventurepinocchio_01_collodi.ogg
```

--------------------------------

### Auto-detect and use best available GPU

Source: https://docs.mozilla.ai/llamafile/whisperfile/gpu.md

Use `--gpu auto` to let whisperfile automatically detect and utilize the most suitable GPU. It falls back to CPU if no compatible GPU is found.

```bash
whisperfile -m models/ggml-medium.en.bin -f audio.wav --gpu auto
```

--------------------------------

### Run llamafile with Ollama Blobs

Source: https://docs.mozilla.ai/llamafile/getting-started/quickstart.md

Execute llamafile using a model directly from Ollama's blob storage by referencing its SHA256 digest. This requires navigating to the blobs directory first. Note that compatibility may vary.

```shell
cd ~/.ollama/models/blobs
llamafile -m sha256-00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29
```

--------------------------------

### Build Vulkan DLL

Source: https://docs.mozilla.ai/llamafile/using-llamafile/building_dlls.md

Executes the build process for the Vulkan DLL. This is typically faster than CUDA or ROCm builds and does not require a parallel script.

```bash
llamafile\vulkan.bat
```

--------------------------------

### Run Self-Contained Whisperfile

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Run the fully self-contained whisperfile executable with only the audio file as an argument, as default arguments are now baked in.

```bash
./whisper-tiny -f whisper.cpp/samples/jfk.wav
```

--------------------------------

### Build CUDA DLL (Parallel)

Source: https://docs.mozilla.ai/llamafile/using-llamafile/building_dlls.md

Initiates the parallel build process for the CUDA DLL. This command should be run from the llamafile directory and may take a significant amount of time.

```bash
cd c:\Users\Your_Username\Workspace\llamafile
llamafile\cuda_parallel.bat
```

--------------------------------

### Build zipalign

Source: https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles.md

Builds the zipalign tool, which is used for bundling files into the llamafile executable. This is a prerequisite for creating llamafiles.

```shell
make o//third_party/zipalign
```

--------------------------------

### Download Large Whisper Model

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Download the large v3 whisper model, which offers the best accuracy but is the slowest to process audio.

```bash
curl -L -o models/whisper-large-v3.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3.bin
```

--------------------------------

### Run Llamafile with a Model

Source: https://docs.mozilla.ai/llamafile/using-llamafile/source_installation.md

Execute the compiled Llamafile with a specified GGUF model.

```shell
./o/llamafile/llamafile --model <gguf_model>
```

--------------------------------

### Running llamafile in CLI mode for prose composition

Source: https://docs.mozilla.ai/llamafile/using-llamafile/running_llamafile.md

Use the --cli argument to run a model in command-line interface mode. The -p argument specifies the prompt for prose generation.

```sh
./Apertus-8B-Instruct-2509.llamafile --cli -p 'Write a story about llamas'
```

--------------------------------

### Register APE file format for Linux

Source: https://docs.mozilla.ai/llamafile/reference/troubleshooting.md

On some Linux systems, you might encounter errors related to `run-detectors` or WINE due to `binfmt_misc` registrations. This snippet adds an additional registration for the APE file format used by llamafile.

```shell
sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo chmod +x /usr/bin/ape
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
```

--------------------------------

### Text-only llamafile arguments

Source: https://docs.mozilla.ai/llamafile/using-llamafile/creating_llamafiles.md

Defines the default arguments for a text-only llamafile, including the model path, quantization settings, and generation parameters. The '/zip/' prefix is required for files packaged within the llamafile.

```shell
-m
/zip/Qwen3-0.6B-Q8_0.gguf
-fa
on
--temp
0.6
--top-k
20
--top-p
0.95
--min-p
0
--presence-penalty
1.5
-c
40960
-n
32768
--no-context-shift
--no-mmap
...
```

--------------------------------

### Embed Model Weights

Source: https://docs.mozilla.ai/llamafile/whisperfile/packaging.md

Embed the downloaded model weights into the whisperfile executable using the `zipalign` tool. The `-0` flag disables DEFLATE compression, which is recommended for binary files.

```bash
o//third_party/zipalign/zipalign -0 whisper-tiny ggml-tiny.en-q5_1.bin
```

--------------------------------

### POST /load

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Loads a different Whisper model at runtime. The model file path is provided in the request body.

```APIDOC
## POST /load

### Description
Load a different model at runtime.

### Method
POST

### Endpoint
/load

### Parameters
#### Request Body
- **model** (file) - Required - Path to the Whisper model weights file.

### Request Example
```bash
curl http://localhost:8080/load \
  -F "model=/path/to/model.bin"
```

### Response
#### Success Response (200)
- **message** (string) - Confirmation message indicating successful model load.

#### Response Example
```
Load was successful!
```
```

--------------------------------

### Clone Llamafile Repository

Source: https://docs.mozilla.ai/llamafile/using-llamafile/building_dlls.md

Clones the Llamafile repository from GitHub into the current workspace directory.

```bash
git clone https://github.com/mozilla-ai/llamafile
```

--------------------------------

### Translate Audio to English

Source: https://docs.mozilla.ai/llamafile/whisperfile/translate.md

Execute Whisperfile with the translation flag (`-tr`) to transcribe and translate the provided audio file into English. This requires a multilingual model.

```bash
o//whisperfile/whisperfile -m ggml-medium-q5_0.bin -f avventurepinocchio_01_collodi.ogg -tr
```

--------------------------------

### POST /inference

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Transcribes an audio file using the whisper model. The audio file should be sent as multipart/form-data.

```APIDOC
## POST /inference

### Description
Transcribe an audio file. Send as multipart/form-data with the audio file in a field named "file".

### Method
POST

### Endpoint
/inference

### Parameters
#### Request Body
- **file** (file) - Required - The audio file to transcribe.
- **response_format** (string) - Optional - Output format: json, text, srt, vtt, verbose_json (default: json)
- **language** (string) - Optional - Spoken language or 'auto' for detection
- **translate** (string) - Optional - Set to 'true' to translate to English
- **temperature** (number) - Optional - Sampling temperature
- **prompt** (string) - Optional - Initial prompt for the model

### Request Example
```bash
curl http://localhost:8080/inference \
  -F "file=@whisper.cpp/samples/jfk.wav" \
  -F "response_format=json"
```

### Response
#### Success Response (200)
- **text** (string) - The transcribed text from the audio file.

#### Response Example
```json
{
  "text": "And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country."
}
```
```

--------------------------------

### Download Medium Whisper Model

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Download the medium-sized whisper model for improved accuracy, suitable for accurately decoding complex text. Note that this model is slower than the tiny model.

```bash
curl -LO https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium.en.bin
```

--------------------------------

### Suppress GPU backend warnings

Source: https://docs.mozilla.ai/llamafile/whisperfile/gpu.md

Benign warnings about missing GPU backend libraries can be suppressed by redirecting stderr to `/dev/null`.

```bash
whisperfile -m models/ggml-medium.en.bin -f audio.wav 2>/dev/null
```

--------------------------------

### Disable GPU acceleration

Source: https://docs.mozilla.ai/llamafile/whisperfile/gpu.md

To completely turn off GPU acceleration and force CPU usage, use the `--no-gpu` flag.

```bash
whisperfile -m models/ggml-medium.en.bin -f audio.wav --no-gpu
```

--------------------------------

### Check Server Health

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Use this endpoint to check if the whisper-server is running and ready to accept requests. It returns an HTTP 503 if the model is still loading.

```bash
curl http://localhost:8080/health
```

--------------------------------

### Transcribe Audio with Medium Model

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Transcribe an audio file using the medium whisper model for higher accuracy. This command uses the downloaded medium model and suppresses verbose output.

```bash
o//whisperfile/whisperfile -m ggml-medium.en.bin -f raven_poe_64kb.mp3 --no-prints
```

--------------------------------

### Transcribe Audio File

Source: https://docs.mozilla.ai/llamafile/whisperfile/server.md

Send an audio file using multipart/form-data to the /inference endpoint for transcription. Optional parameters can be included to control output format, language, and translation.

```bash
curl http://localhost:8080/inference \
  -F "file=@whisper.cpp/samples/jfk.wav" \
  -F "response_format=json"
```

--------------------------------

### Transcribe Audio with Large Model

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Transcribe an audio file using the large v3 whisper model for maximum accuracy. This command uses the downloaded large model and suppresses verbose output.

```bash
o//whisperfile/whisperfile -m models/whisper-large-v3.bin -f raven_poe_64kb.mp3 --no-prints
```

--------------------------------

### Transcribe MP3 Audio with Color Output

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Transcribe an MP3 audio file using whisperfile. The -pc flag enables color-coded terminal output to indicate transcription confidence.

```bash
o//whisperfile/whisperfile -m models/whisper-tiny.en-q5_1.bin -f raven_poe_64kb.mp3 -pc
```

--------------------------------

### Permanently disable WSL interop in wsl.conf

Source: https://docs.mozilla.ai/llamafile/reference/troubleshooting.md

For persistent disabling of WSL interop, add this configuration to `/etc/wsl.conf`. This is an alternative to using the CLI commands.

```ini
[interop]
enabled=false
```

--------------------------------

### Transcribe JFK Speech Audio

Source: https://docs.mozilla.ai/llamafile/whisperfile/getting-started.md

Transcribe a .wav audio file using the compiled whisperfile software and the tiny model. The --no-prints flag can be used to reduce verbose logging.

```bash
o//whisperfile/whisperfile -m models/whisper-tiny.en-q5_1.bin whisperfile/jfk.wav --no-prints
```

--------------------------------

### Override Source Language for Translation

Source: https://docs.mozilla.ai/llamafile/whisperfile/translate.md

Specify the source language using the `-l` flag when auto-detection might fail, such as in recordings with mixed languages. This ensures accurate translation from the intended source.

```bash
o//whisperfile/whisperfile -m ggml-medium-q5_0.bin -f audio.ogg -tr -l fr
```

--------------------------------

### Disable WSL interop feature (late)

Source: https://docs.mozilla.ai/llamafile/reference/troubleshooting.md

In Windows 11 with WSL 2, the location of the interop flag has changed. This command targets the 'late' interop flag, which may be required instead or additionally.

```shell
sudo sh -c "echo -1 > /proc/sys/fs/binfmt_misc/WSLInterop-late"
```

--------------------------------

### Disable WSL interop feature

Source: https://docs.mozilla.ai/llamafile/reference/troubleshooting.md

Disables the WIN32 interop feature in WSL to potentially resolve issues. This command directly modifies the kernel's binfmt_misc settings.

```shell
sudo sh -c "echo -1 > /proc/sys/fs/binfmt_misc/WSLInterop"
```
Tokens:
Raw