### Quickstart TTS Example

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/tts/README.md

Run the TTS example directly. Requires llama.cpp to be built with SSL support. Automatically downloads models.

```console
$ build/bin/llama-tts --tts-oute-default -p "Hello world" && aplay output.wav
```

--------------------------------

### Setup Python Virtual Environment

Source: https://github.com/ggml-org/llama.cpp/blob/master/examples/model-conversion/README.md

Create and activate a Python virtual environment and install dependencies.

```console
python3.11 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

--------------------------------

### Install Dependencies and Run Dev Server

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README-dev.md

Install Node.js dependencies and start the SvelteKit development server with hot reloading. Ensure llama-server is running on port 8080.

```sh
cd tools/ui
npm i

# run dev server (with hot reload)
npm run dev
```

--------------------------------

### Start llama-server for Benchmarking

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/bench/speed-bench/README.md

Launches the llama.cpp server with essential parameters for benchmarking. Adjust `--np` to match the client's `--concurrency` for accurate throughput measurements. Includes example flags for model loading and speculative decoding.

```bash
llama-server \
  -m target.gguf \
  -c 8192 \
  --port 8080 \
  -ngl 99 -fa on \
  --np 1 \
  --jinja
```

--------------------------------

### Example nvcc Version Output

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/CUDA-FEDORA.md

This is an example output from the 'nvcc --version' command, indicating a successful CUDA installation and the specific version details.

```text
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0
```

--------------------------------

### Install and Verify clinfo

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md

Install the 'clinfo' utility to verify OpenCL installation and list available OpenCL devices. Use 'clinfo -l' to display the detected platforms and devices.

```bash
sudo apt install clinfo
sudo clinfo -l
```

--------------------------------

### Install k6 and SSE Extension

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/bench/README.md

Installs the k6 load testing tool and builds it with the xk6-sse extension for Server-Sent Events support. Requires Go to be installed.

```shell
go install go.k6.io/xk6/cmd/xk6@latest
$GOPATH/bin/xk6 build master \
--with github.com/phymbert/xk6-sse
```

--------------------------------

### Multimodal CLI for Image Description

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal/MobileVLM.md

This example demonstrates using the MobileVLM CLI to get a description of an image. It's similar to the previous example but with a different prompt focused on describing the image content.

```shell
./llama-mtmd-cli \
    -m /data/local/tmp/ggml-model-q4_k.gguf \
    --mmproj /data/local/tmp/mmproj-model-f16.gguf \
    -p "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat is in the image? ASSISTANT:" \
    --n-gpu-layers 999
```

--------------------------------

### Run Summary Request with OLMoE-1B-7B on Two Hexagon NPUs

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md

This example demonstrates running a summary request for the large OLMoE-1B-7B model, which requires two Hexagon NPU sessions (HTP0 and HTP1). It highlights the setup for multiple devices and the resulting logs, including model offloading and detailed memory breakdown for each NPU.

```bash
~/src/llama.cpp$ M=OLMoE-1B-7B-0125-Instruct-Q4_0.gguf NDEV=2 D=HTP0,HTP1 ./scripts/snapdragon/adb/run-completion.sh -f surfing.txt
... 
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v81
ggml-hex: allocating new session: HTP0
ggml-hex: allocating new session: HTP1
... 
load_tensors: offloading output layer to GPU
load_tensors: offloaded 17/17 layers to GPU
load_tensors:          CPU model buffer size =   143.86 MiB
load_tensors:         HTP1 model buffer size =     0.23 MiB
load_tensors:  HTP1-REPACK model buffer size =  1575.00 MiB
load_tensors:         HTP0 model buffer size =     0.28 MiB
load_tensors:  HTP0-REPACK model buffer size =  2025.00 MiB
... 
llama_context:        CPU  output buffer size =     0.19 MiB
llama_kv_cache:       HTP1 KV buffer size =   238.00 MiB
llama_kv_cache:       HTP0 KV buffer size =   306.00 MiB
llama_kv_cache: size =  544.00 MiB (  8192 cells,  16 layers,  1/1 seqs), K (q8_0):  272.00 MiB, V (q8_0):  272.00 MiB
llama_context:       HTP0 compute buffer size =    15.00 MiB
llama_context:       HTP1 compute buffer size =    15.00 MiB
llama_context:        CPU compute buffer size =    24.56 MiB
... 
llama_perf_context_print: prompt eval time =    1730.57 ms /   212 tokens (    8.16 ms per token,   122.50 tokens per second)
llama_perf_context_print:        eval time =    5624.75 ms /   257 runs   (   21.89 ms per token,    45.69 tokens per second)
llama_perf_context_print:       total time =    7377.33 ms /   469 tokens
llama_perf_context_print:    graphs reused =        255
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - HTP0 (Hexagon)     |  2048 = 2048 + (   0 =     0 +       0 +       0) +           0 |
llama_memory_breakdown_print: |   - HTP1 (Hexagon)     |  2048 = 2048 + (   0 =     0 +       0 +       0) +           0 |
llama_memory_breakdown_print: |   - Host               |                  742 =   144 +     544 +      54                |
llama_memory_breakdown_print: |   - HTP1-REPACK        |                 1575 =  1575 +       0 +       0                |
llama_memory_breakdown_print: |   - HTP0-REPACK        |                 2025 =  2025 +       0 +       0                |
```

--------------------------------

### Install Dependencies

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/ui/README.md

Navigate to the ui directory and install project dependencies using npm.

```bash
cd tools/ui
npm install
```

--------------------------------

### Run Simple Chat Example

Source: https://github.com/ggml-org/llama.cpp/blob/master/examples/simple-chat/README.md

Execute the simple chat example with a specified model and context size.

```bash
./llama-simple-chat -m Meta-Llama-3.1-8B-Instruct.gguf -c 2048
...
```

--------------------------------

### Create Installable Package

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/linux.md

Generate an installable package for llama.cpp by specifying an installation prefix and then zipping the contents.

```bash
[d]/workspace> cmake --install build-snapdragon --prefix pkg-snapdragon
[d]/workspace> zip -r pkg-snapdragon.zip pkg-snapdragon
```

--------------------------------

### Start Development Server

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/ui/README.md

Starts the Vite development server for the UI frontend and Storybook for component documentation.

```bash
npm run dev
```

--------------------------------

### Run Completion with Llama-3.2-1B on Hexagon NPU

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md

This example shows how to run a completion request for the Llama-3.2-1B model on a Snapdragon device using the Hexagon NPU (HTP0). It demonstrates the setup and output logs, including model offloading to the GPU and memory usage.

```bash
~/src/llama.cpp$ M=Llama-3.2-1B-Instruct-Q4_0.gguf D=HTP0 ./scripts/snapdragon/adb/run-completion.sh -p "what is the most popular cookie in the world?"
... 
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v79
ggml-hex: allocating new session: HTP0
ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v79.so?htp_iface_skel_handle_invoke&_modver=1.0&_dom=cdsp&_session=0 handle 0xb4000072c7955e50
... 
load_tensors: offloading output layer to GPU
load_tensors: offloaded 17/17 layers to GPU
load_tensors:          CPU model buffer size =   225.49 MiB
load_tensors:         HTP0 model buffer size =     0.26 MiB
load_tensors:  HTP0-REPACK model buffer size =   504.00 MiB
... 
I hope this helps you understand the world's most popular cookies! [end of text] 
... 
llama_perf_sampler_print:    sampling time =      30.08 ms /   487 runs   (    0.06 ms per token, 16191.77 tokens per second)
llama_perf_context_print:        load time =     617.94 ms
llama_perf_context_print: prompt eval time =      80.76 ms /    11 tokens (    7.34 ms per token,   136.21 tokens per second)
llama_perf_context_print:        eval time =    9210.59 ms /   475 runs   (   19.39 ms per token,    51.57 tokens per second)
llama_perf_context_print:       total time =    9454.92 ms /   486 tokens
llama_perf_context_print:    graphs reused =        473
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - HTP0 (Hexagon)     |  2048 = 2048 + (   0 =     0 +       0 +       0) +           0 |
llama_memory_breakdown_print: |   - Host               |                  439 =   225 +     136 +      77                |
llama_memory_breakdown_print: |   - HTP0-REPACK        |                  504 =   504 +       0 +       0                |
```

--------------------------------

### Start llama-server (MODEL mode)

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/ui/README.md

Starts the llama-server in single-model mode. Ensure you have a model file available.

```bash
./llama-server -m model.gguf
```

--------------------------------

### Install Build and Twine for Publishing

Source: https://github.com/ggml-org/llama.cpp/blob/master/gguf-py/README.md

Install the 'build' and 'twine' packages, which are required for manually building and uploading Python packages.

```sh
pip install build twine
```

--------------------------------

### Start Server with Template Override for Firefunction

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md

Start the server with the specified chat template file for the firefunction model.

```shell
llama-server --jinja -fa -hf bartowski/firefunction-v2-GGUF -hff firefunction-v2-IQ1_M.gguf \
    --chat-template-file models/templates/fireworks-ai-llama-3-firefunction-v2.jinja
```

--------------------------------

### Start llama-server (ROUTER mode)

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/ui/README.md

Starts the llama-server in multi-model router mode. Specify the directory containing your models.

```bash
./llama-server --models-dir /path/to/models
```

--------------------------------

### Example Complete Configuration for macOS with Metal

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/VirtGPU/configuration.md

This example shows how to configure the VirtGPU hypervisor and backend for a macOS host using the Metal backend. It sets environment variables for library paths, registration functions, logging, and guest configuration.

```bash
# Hypervisor environment
export VIRGL_APIR_BACKEND_LIBRARY="/opt/llama.cpp/lib/libggml-virtgpu-backend.dylib"

# Backend configuration
export APIR_LLAMA_CPP_GGML_LIBRARY_PATH="/opt/llama.cpp/lib/libggml-metal.dylib"
export APIR_LLAMA_CPP_GGML_LIBRARY_REG="ggml_backend_metal_reg"

# Optional logging
export VIRGL_APIR_LOG_TO_FILE="/tmp/apir.log"
export APIR_LLAMA_CPP_LOG_TO_FILE="/tmp/ggml.log"

# Guest configuration
export GGML_REMOTING_USE_APIR_CAPSET=1
```

--------------------------------

### Set up oneAPI Environment and List SYCL Devices

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md

Source the oneAPI environment script to enable SYCL and related tools. Use 'sycl-ls' to list all available SYCL devices on the system.

```bash
source /opt/intel/oneapi/setvars.sh
sycl-ls
```

--------------------------------

### Install OpenCL with vcpkg on Windows

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md

Sets up vcpkg and installs the OpenCL library for Windows development. Includes optional integration with Visual Studio.

```powershell
# Windows PowerShell
cd C:\
git clone https://github.com/microsoft/vcpkg
cd vcpkg
.\bootstrap-vcpkg.bat
.\vcpkg install opencl
# Optional but recommended: Integrate vcpkg with Visual Studio / CMake:
.\vcpkg integrate install
```

--------------------------------

### Install OpenCL Headers and Library on Windows (PowerShell)

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENCL.md

Installs OpenCL headers and the ICD loader using CMake and Ninja on Windows. This setup is required for building llama.cpp with OpenCL on Windows Arm64.

```powershell
mkdir -p ~/dev/llm

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers
mkdir build && cd build
cmake .. -G Ninja \
  -DBUILD_TESTING=OFF \
  -DOPENCL_HEADERS_BUILD_TESTING=OFF \
  -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF \
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader
mkdir build && cd build
cmake .. -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install
```

--------------------------------

### Verify OpenVINO Environment Variable

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md

Checks if the OpenVINO environment variable is set, indicating a successful installation or setup.

```bash
echo $OpenVINO_DIR
```

--------------------------------

### Run Retrieval Example

Source: https://github.com/ggml-org/llama.cpp/blob/master/examples/retrieval/README.md

This command demonstrates how to run the retrieval example, specifying the model, top-k value, context files to embed, chunk size, and chunk separator.

```bash
llama-retrieval --model ./models/bge-base-en-v1.5-f16.gguf --top-k 3 --context-file README.md --context-file License --chunk-size 100 --chunk-separator .
```

--------------------------------

### Model Preset Configuration Example

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Example INI file demonstrating global settings, model-specific configurations, and custom model paths.

```ini
version = 1

; (Optional) This section provides global settings shared across all presets.
; If the same key is defined in a specific preset, it will override the value in this global section.
[*]
c = 8192
n-gpu-layers = 8

; If the key corresponds to an existing model on the server,
; this will be used as the default config for that model
[ggml-org/MY-MODEL-GGUF:Q8_0]
; string value
chat-template = chatml
; numeric value
n-gpu-layers = 123
; flag value (for certain flags, you need to use the "no-" prefix for negation)
jinja = true
; shorthand argument (for example, context size)
c = 4096
; environment variable name
LLAMA_ARG_CACHE_RAM = 0
; file paths are relative to server's CWD
model-draft = ./my-models/draft.gguf
; but it's RECOMMENDED to use absolute path
model-draft = /Users/abc/my-models/draft.gguf

; If the key does NOT correspond to an existing model,
; you need to specify at least the model path or HF repo
[custom_model]
model = /Users/abc/my-awesome-model-Q4_K_M.gguf
```

--------------------------------

### Get Current Weather Tool

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md

Example of calling the API with a 'get_current_weather' tool. This demonstrates how to define a tool with parameters for location.

```APIDOC
## POST /v1/chat/completions

### Description
Retrieves the current weather for a specified location using the `get_current_weather` tool.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Request Body
- **model** (string) - Required - The model to use for the chat completion.
- **messages** (array) - Required - A list of message objects representing the conversation history.
  - **role** (string) - Required - The role of the message sender (e.g., `system`, `user`).
  - **content** (string) - Required - The content of the message.
- **tools** (array) - Required - A list of tools the model can use. Each tool should have a `type` and `function` definition.
  - **type** (string) - Required - The type of tool, e.g., `function`.
  - **function** (object) - Required - The definition of the function tool.
    - **name** (string) - Required - The name of the function (e.g., `get_current_weather`).
    - **description** (string) - Required - A description of what the function does.
    - **parameters** (object) - Required - The parameters the function accepts.
      - **type** (string) - Required - The schema type, usually `object`.
      - **properties** (object) - Required - An object defining the properties of the parameters.
        - **location** (string) - Required - The city and country/state for which to get the weather (e.g., `San Francisco, CA`, or `Paris, France`).
      - **required** (array) - Required - A list of required parameter names.
        - **location** (string)

### Request Example
```json
{
    "model": "gpt-3.5-turbo",
    "messages": [
        {"role": "system", "content": "You are a chatbot that uses tools/functions. Dont overthink things."},
        {"role": "user", "content": "What is the weather in Istanbul?"}
    ],
    "tools": [{
        "type":"function",
        "function":{
            "name":"get_current_weather",
            "description":"Get the current weather in a given location",
            "parameters":{
                "type":"object",
                "properties":{
                    "location":{
                        "type":"string",
                        "description":"The city and country/state, e.g. `San Francisco, CA`, or `Paris, France`"
                    }
                },
                "required":["location"]
            }
        }
    }]
}
```
```

--------------------------------

### Setup Python Environment for TTS Client

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/tts/README.md

Creates a virtual environment and installs necessary Python packages (requests, numpy) for the TTS client script.

```console
$ python3 -m venv venv
$ source venv/bin/activate
(venv) pip install requests numpy
```

--------------------------------

### Set up LunarG Vulkan SDK on macOS

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md

Source the Vulkan SDK setup script to configure your environment. This is typically done after installing the LunarG Vulkan SDK.

```bash
source /path/to/vulkan-sdk/setup-env.sh
```

--------------------------------

### Example Response for GET /slots

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

This JSON output shows the state of two processing slots. It includes details about task ID, context size, processing status, and sampling parameters for each slot.

```json
[
  {
    "id": 0,
    "id_task": 135,
    "n_ctx": 65536,
    "speculative": false,
    "is_processing": true,
    "params": {
      "n_predict": -1,
      "seed": 4294967295,
      "temperature": 0.800000011920929,
      "dynatemp_range": 0.0,
      "dynatemp_exponent": 1.0,
      "top_k": 40,
      "top_p": 0.949999988079071,
      "min_p": 0.05000000074505806,
      "top_n_sigma": -1.0,
      "xtc_probability": 0.0,
      "xtc_threshold": 0.10000000149011612,
      "typical_p": 1.0,
      "repeat_last_n": 64,
      "repeat_penalty": 1.0,
      "presence_penalty": 0.0,
      "frequency_penalty": 0.0,
      "dry_multiplier": 0.0,
      "dry_base": 1.75,
      "dry_allowed_length": 2,
      "dry_penalty_last_n": 131072,
      "mirostat": 0,
      "mirostat_tau": 5.0,
      "mirostat_eta": 0.10000000149011612,
      "max_tokens": -1,
      "n_keep": 0,
      "n_discard": 0,
      "ignore_eos": false,
      "stream": true,
      "n_probs": 0,
      "min_keep": 0,
      "chat_format": "GPT-OSS",
      "reasoning_format": "none",
      "reasoning_in_content": false,
      "generation_prompt": "",
      "samplers": [
        "penalties",
        "dry",
        "top_k",
        "typ_p",
        "top_p",
        "min_p",
        "xtc",
        "temperature"
      ],
      "speculative.n_max": 16,
      "speculative.n_min": 0,
      "speculative.p_min": 0.75,
      "timings_per_token": false,
      "post_sampling_probs": false,
      "lora": []
    },
    "next_token": {
      "has_next_token": true,
      "has_new_line": false,
      "n_remain": -1,
      "n_decoded": 0
    }
  },
  {
    "id": 1,
    "id_task": 0,
    "n_ctx": 65536,
    "speculative": false,
    "is_processing": true,
    "params": {
      "n_predict": -1,
      "seed": 4294967295,
      "temperature": 0.800000011920929,
      "dynatemp_range": 0.0,
      "dynatemp_exponent": 1.0,
      "top_k": 40,
      "top_p": 0.949999988079071,
      "min_p": 0.05000000074505806,
      "top_n_sigma": -1.0,
      "xtc_probability": 0.0,
      "xtc_threshold": 0.10000000149011612,
      "typical_p": 1.0,
      "repeat_last_n": 64,
      "repeat_penalty": 1.0,
      "presence_penalty": 0.0,
      "frequency_penalty": 0.0,
      "dry_multiplier": 0.0,
      "dry_base": 1.75,
      "dry_allowed_length": 2,
      "dry_penalty_last_n": 131072,
      "mirostat": 0,
      "mirostat_tau": 5.0,
      "mirostat_eta": 0.10000000149011612,
      "max_tokens": -1,
      "n_keep": 0,
      "n_discard": 0,
      "ignore_eos": false,
      "stream": true,
      "n_probs": 0,
      "min_keep": 0,
      "chat_format": "GPT-OSS",
      "reasoning_format": "none",
      "reasoning_in_content": false,
      "generation_prompt": "",
      "samplers": [
        "penalties",
        "dry",
        "top_k",
        "typ_p",
        "top_p",
        "min_p",
        "xtc",
        "temperature"
      ],
      "speculative.n_max": 16,
      "speculative.n_min": 0,
      "speculative.p_min": 0.75,
      "timings_per_token": false,
      "post_sampling_probs": false,
      "lora": []
    },
    "next_token": {
      "has_next_token": true,
      "has_new_line": true,
      "n_remain": -1,
      "n_decoded": 136
    }
  }
]
```

--------------------------------

### Replay or Live Reattach Stream

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README-dev.md

Clients can reattach to a live stream or replay past events using a GET request to `/v1/stream/:conv_id`. The `from` query parameter specifies the byte offset to start replaying from.

```APIDOC
## GET /v1/stream/:conv_id

### Description
Replays buffered stream events or connects to a live stream for a given conversation.

### Method
GET

### Endpoint
/v1/stream/:conv_id

### Parameters
#### Path Parameters
- **conv_id** (string) - Required - The conversation ID for the stream.

#### Query Parameters
- **from** (integer) - Optional - The byte offset from which to start replaying the stream. If the offset is invalid or before the current buffer, a 400 error is returned.

### Response
- **Content-Type**: text/event-stream

#### Success Response (200)
- Streams Server-Sent Events (SSE) data.

#### Error Response (400)
- Returned if the `from` query parameter is invalid or refers to data that has already been dropped from the buffer.
```

--------------------------------

### CMake Build Configuration for Retrieval Example

Source: https://github.com/ggml-org/llama.cpp/blob/master/examples/retrieval/CMakeLists.txt

This snippet shows the CMakeLists.txt configuration for building the retrieval executable. It defines the target, adds the source file, specifies installation rules, links necessary libraries, and sets the C++ standard.

```cmake
set(TARGET llama-retrieval)
add_executable(${TARGET} retrieval.cpp)
install(TARGETS ${TARGET} RUNTIME)
target_link_libraries(${TARGET} PRIVATE llama-common llama ${CMAKE_THREAD_LIBS_INIT})
target_compile_features(${TARGET} PRIVATE cxx_std_17)
```

--------------------------------

### Setup Debug Build Context

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/development/debugging-tests.md

Prepare a clean directory for building and debugging. This ensures a consistent environment and avoids interference from previous builds.

```bash
rm -rf build-ci-debug && mkdir build-ci-debug && cd build-ci-debug
```

--------------------------------

### Start Server with Native Support

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md

Use this command to start the llama.cpp server with native support for models that have a tool-aware chat template.

```shell
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
llama-server --jinja -fa -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K_L
llama-server --jinja -fa -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M
llama-server --jinja -fa -hf ibm-granite/granite-4.1-3b-GGUF:Q4_K_M
```

--------------------------------

### Install llama.cpp Package

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md

Generate an installable package for llama.cpp by running the CMake install command. This installs libraries and binaries to a specified prefix.

```bash
[d]/workspace> cmake --install build-snapdragon --prefix pkg-snapdragon/llama.cpp
-- Install configuration: "Release"
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-cpu.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-opencl.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-hexagon.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-htp-v73.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-htp-v75.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-htp-v79.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml-htp-v81.so
-- Installing: /workspace/pkg-snapdragon/llama.cpp/lib/libggml.so
...
-- Installing: /workspace/pkg-snapdragon/llama.cpp/bin/llama-bench
-- Installing: /workspace/pkg-snapdragon/llama.cpp/bin/llama-cli
...

```

--------------------------------

### Simple Server Usage with -hf

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md

Shows how to start the llama.cpp server with the -hf flag for multimodal model support.

```shell
llama-server -hf ggml-org/gemma-3-4b-it-GGUF
```

--------------------------------

### Install Compiled Metal Library

Source: https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-metal/CMakeLists.txt

Installs the compiled default.metallib to the binary directory during installation.

```cmake
install(
            FILES ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/default.metallib
            DESTINATION ${CMAKE_INSTALL_BINDIR}
        )
```

--------------------------------

### Run LLaVA CLI Example (LLaVA 1.6)

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal/llava.md

Example command to run the `llama-mtmd-cli` binary with LLaVA 1.6 models. Note the different mmproj path.

```console
./llama-mtmd-cli -m ../llava-v1.6-vicuna-7b/ggml-model-f16.gguf --mmproj vit/mmproj-model-f16.gguf
```

--------------------------------

### Download Sample Model for Testing

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md

Instructions for downloading a sample Llama-3.2-1B-Instruct-Q4_K_M.gguf model. Examples are provided for Linux, Windows PowerShell, and Windows Command Line.

```bash
# Linux
mkdir -p ~/models/
wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
     -O ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
```

```powershell
# Windows PowerShell
mkdir C:\models
Invoke-WebRequest -Uri https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -OutFile C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf
```

```bash
# Windows Command Line
mkdir C:\models
curl -L https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -o C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf
```

--------------------------------

### Prepare Visual Encoder Directory

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal/granitevision.md

Create a directory for the visual encoder and copy the llava.clip and llava.projector files into it.

```bash
$ ENCODER_PATH=$PWD/visual_encoder
$ mkdir $ENCODER_PATH

$ cp $GRANITE_MODEL/llava.clip $ENCODER_PATH/pytorch_model.bin
$ cp $GRANITE_MODEL/llava.projector $ENCODER_PATH/
```

--------------------------------

### Install gguf Python Package

Source: https://github.com/ggml-org/llama.cpp/blob/master/gguf-py/README.md

Install the gguf Python package using pip. This is the basic installation command.

```sh
pip install gguf
```

--------------------------------

### Install llama.cpp with Nix

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md

Install llama.cpp on macOS and Linux using Nix. Supports both flake-enabled and non-flake-enabled installs.

```sh
nix profile install nixpkgs#llama-cpp

# For flake enabled installs.

Or

nix-env --file '<nixpkgs>' --install --attr llama-cpp

# For non-flake enabled installs.
```

--------------------------------

### Install Nvidia Driver Libraries (Guest)

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/CUDA-FEDORA.md

Installs the necessary Nvidia driver libraries and related packages on the guest system when libcuda.so.1 is missing.

```bash
sudo dnf install nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced
```

--------------------------------

### GET /v1/models

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Returns information about the loaded model. The returned list always has one single element. The `meta` field can be `null` (for example, while the model is still loading). By default, model `id` field is the path to model file, specified via `-m`. You can set a custom value for model `id` field via `--alias` argument.

```APIDOC
## GET /v1/models

### Description
Returns information about the loaded model. The returned list always has one single element. The `meta` field can be `null` (for example, while the model is still loading).

By default, model `id` field is the path to model file, specified via `-m`. You can set a custom value for model `id` field via `--alias` argument. For example, `--alias gpt-4o-mini`.

### Endpoint
`/v1/models`

### Response
#### Success Response (200)
- **object** (string) - Type of the response, usually "list".
- **data** (array) - An array of model objects.
  - **id** (string) - The model identifier.
  - **object** (string) - The type of the object, usually "model".
  - **created** (integer) - Timestamp of model creation.
  - **owned_by** (string) - The owner of the model, e.g., "llamacpp".
  - **meta** (object | null) - Metadata about the model, can be null.
    - **vocab_type** (integer) - Vocabulary type.
    - **n_vocab** (integer) - Number of vocabulary items.
    - **n_ctx_train** (integer) - Training context size.
    - **n_embd** (integer) - Embedding dimension.
    - **n_params** (integer) - Number of model parameters.
    - **size** (integer) - Size of the model file in bytes.

### Response Example
```json
{
    "object": "list",
    "data": [
        {
            "id": "../models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
            "object": "model",
            "created": 1735142223,
            "owned_by": "llamacpp",
            "meta": {
                "vocab_type": 2,
                "n_vocab": 128256,
                "n_ctx_train": 131072,
                "n_embd": 4096,
                "n_params": 8030261312,
                "size": 4912898304
            }
        }
    ]
}
```
```

--------------------------------

### SQL Schema and Data Example

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md

Example SQL statements showing the table creation schema and sample data insertion for llama-bench results.

```sql
CREATE TABLE IF NOT EXISTS llama_bench (
  build_commit TEXT,
  build_number INTEGER,
  cpu_info TEXT,
  gpu_info TEXT,
  backends TEXT,
  model_filename TEXT,
  model_type TEXT,
  model_size INTEGER,
  model_n_params INTEGER,
  n_batch INTEGER,
  n_ubatch INTEGER,
  n_threads INTEGER,
  cpu_mask TEXT,
  cpu_strict INTEGER,
  poll INTEGER,
  type_k TEXT,
  type_v TEXT,
  n_gpu_layers INTEGER,
  n_cpu_moe INTEGER,
  split_mode TEXT,
  main_gpu INTEGER,
  no_kv_offload INTEGER,
  flash_attn INTEGER,
  devices TEXT,
  tensor_split TEXT,
  tensor_buft_overrides TEXT,
  use_mmap INTEGER,
  use_direct_io INTEGER,
  embeddings INTEGER,
  no_op_offload INTEGER,
  no_host INTEGER,
  fit_target INTEGER,
  fit_min_ctx INTEGER,
  n_prompt INTEGER,
  n_gen INTEGER,
  n_depth INTEGER,
  test_time TEXT,
  avg_ns INTEGER,
  stddev_ns INTEGER,
  avg_ts REAL,
  stddev_ts REAL
);

INSERT INTO llama_bench (build_commit, build_number, cpu_info, gpu_info, backends, model_filename, model_type, model_size, model_n_params, n_batch, n_ubatch, n_threads, cpu_mask, cpu_strict, poll, type_k, type_v, n_gpu_layers, n_cpu_moe, split_mode, main_gpu, no_kv_offload, flash_attn, devices, tensor_split, tensor_buft_overrides, use_mmap, use_direct_io, embeddings, no_op_offload, no_host, fit_target, fit_min_ctx, n_prompt, n_gen, n_depth, test_time, avg_ns, stddev_ns, avg_ts, stddev_ts) VALUES ('8cf427ff', '5163', 'AMD Ryzen 7 7800X3D 8-Core Processor', 'NVIDIA GeForce RTX 4080', 'CUDA', 'models/Qwen2.5-7B-Instruct-Q4_K_M.gguf', 'qwen2 7B Q4_K - Medium', '4677120000', '7615616512', '2048', '512', '8', '0x0', '0', '50', 'f16', 'f16', '-1', '0', 'layer', '0', '0', '-1', 'auto', '0.00', 'none', '1', '0', '0', '0', '0', '0', '0', '512', '0', '0', '2025-04-24T12:00:08Z', '69905000', '519516', '7324.546977', '54.032613');
INSERT INTO llama_bench (build_commit, build_number, cpu_info, gpu_info, backends, model_filename, model_type, model_size, model_n_params, n_batch, n_ubatch, n_threads, cpu_mask, cpu_strict, poll, type_k, type_v, n_gpu_layers, n_cpu_moe, split_mode, main_gpu, no_kv_offload, flash_attn, devices, tensor_split, tensor_buft_overrides, use_mmap, use_direct_io, embeddings, no_op_offload, no_host, fit_target, fit_min_ctx, n_prompt, n_gen, n_depth, test_time, avg_ns, stddev_ns, avg_ts, stddev_ts) VALUES ('8cf427ff', '5163', 'AMD Ryzen 7 7800X3D 8-Core Processor', 'NVIDIA GeForce RTX 4080', 'CUDA', 'models/Qwen2.5-7B-Instruct-Q4_K_M.gguf', 'qwen2 7B Q4_K - Medium', '4677120000', '7615616512', '2048', '512', '8', '0x0', '0', '50', 'f16', 'f16', '-1', '0', 'layer', '0', '0', '-1', 'auto', '0.00', 'none', '1', '0', '0', '0', '0', '0', '0', '0', '128', '0', '2025-04-24T12:00:09Z', '1063608780', '4464130', '120.346696', '0.504647');
```

--------------------------------

### Install Python Requirements

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md

Installs the necessary Python packages for model conversion. Use `uv` for faster installation if available.

```bash
python3 -m pip install -r requirements.txt
```

```bash
uv pip install -r requirements.txt --index-strategy unsafe-best-match
```

--------------------------------

### Server Usage with Local Model and MMProj

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md

Illustrates starting the llama.cpp server using local model and multimodal projector files.

```shell
llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf
```

--------------------------------

### Upgrade Pip for Editable Installation

Source: https://github.com/ggml-org/llama.cpp/blob/master/gguf-py/README.md

If an editable installation requires an upgraded Pip, use this command to install the latest version.

```sh
pip install --upgrade pip
```

--------------------------------

### Verify CUDA Installation

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/CUDA-FEDORA.md

Checks the installed version of the NVIDIA CUDA Compiler (nvcc) to confirm that CUDA has been successfully installed and configured.

```bash
nvcc --version
```

--------------------------------

### CMakeLists.txt for Batched Example

Source: https://github.com/ggml-org/llama.cpp/blob/master/examples/batched/CMakeLists.txt

Configures the build for the batched executable, linking common and llama libraries, and setting C++17 standard.

```cmake
set(TARGET llama-batched)
add_executable(${TARGET} batched.cpp)
install(TARGETS ${TARGET} RUNTIME)
target_link_libraries(${TARGET} PRIVATE llama-common llama ${CMAKE_THREAD_LIBS_INIT})
target_compile_features(${TARGET} PRIVATE cxx_std_17)
```

--------------------------------

### Conditional Installation of Target

Source: https://github.com/ggml-org/llama.cpp/blob/master/app/CMakeLists.txt

Installs the 'llama-app' target as a runtime executable only if the 'LLAMA_TOOLS_INSTALL' build option is enabled. This allows for selective installation of tools.

```cmake
if(LLAMA_TOOLS_INSTALL)
    install(TARGETS ${TARGET} RUNTIME)
endif()
```

--------------------------------

### Windows Batch Script for OpenVINO Setup and Build

Source: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md

This script automates the setup of OpenVINO, Visual Studio Build Tools, and the compilation of llama.cpp with OpenVINO support using CMake and Ninja. It handles directory creation, environment variable setup, and build cache cleaning.

```batch
REM Move the single top-level folder contents into the versioned install dir.
REM NOTE: delayed expansion (!VAR!) is required because the surrounding else( ... )
REM block is parsed once up-front, so %OPENVINO_EXTRACTED% would expand to ""
REM and xcopy would then treat "\*" as C:\* and fail with "Cannot perform a cyclic copy".
set "OPENVINO_EXTRACTED="
for /d %%i in ("%OPENVINO_EXTRACT_TMP%\*") do set "OPENVINO_EXTRACTED=%%i"
if not defined OPENVINO_EXTRACTED (
    echo ERROR: Could not locate extracted OpenVINO folder under "%OPENVINO_EXTRACT_TMP%".
    exit /b 1
)
if not exist "%OPENVINO_INSTALL_DIR%" mkdir "%OPENVINO_INSTALL_DIR%"
xcopy /e /i /y /q "!OPENVINO_EXTRACTED!\*" "%OPENVINO_INSTALL_DIR%\" >nul
if errorlevel 1 (
    echo ERROR: Failed to copy OpenVINO from "!OPENVINO_EXTRACTED!" to "%OPENVINO_INSTALL_DIR%".
    echo Re-run this script from an elevated Command Prompt ^(Run as administrator^) if access is denied.
    exit /b 1
)

rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
del "%OPENVINO_ZIP%"
)

REM Refresh junction: C:\Intel\openvino -> C:\Intel\openvino_<version>.
REM `mklink /J` creates a directory junction (no admin / Developer Mode required).
if exist "%OPENVINO_LINK_DIR%" rmdir "%OPENVINO_LINK_DIR%"
mklink /J "%OPENVINO_LINK_DIR%" "%OPENVINO_INSTALL_DIR%" >nul
if errorlevel 1 (
    echo ERROR: Failed to create junction "%OPENVINO_LINK_DIR%" -^> "%OPENVINO_INSTALL_DIR%".
    echo If "%OPENVINO_LINK_DIR%" already exists as a regular non-empty folder, remove it manually and re-run.
    exit /b 1
)

set "OPENVINO_ROOT=%OPENVINO_LINK_DIR%"
echo OpenVINO Ready: %OPENVINO_ROOT% -^> %OPENVINO_INSTALL_DIR%



echo ============================================

echo Setting up compiler environment...
echo ============================================
REM Locate Visual Studio Build Tools vcvars64.bat
set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
if exist "%VSWHERE%" (
    for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products Microsoft.VisualStudio.Product.BuildTools -property installationPath`) do (
        set "VS_PATH=%%i"
    )
)
if defined VS_PATH (
    call "%VS_PATH%\VC\Auxiliary\Build\vcvars64.bat" >nul
) else (
    echo WARNING: Visual Studio Build Tools not found. Compiler may be missing.
)

REM ============================================
REM Clean old build cache
REM ============================================
if exist "build\ReleaseOV" (
    echo Removing old build directory ...
    rmdir /s /q "build\ReleaseOV"
)

echo ============================================

echo Configuring with CMake...
echo ============================================
call "%OPENVINO_ROOT%\setupvars.bat" >nul 2>nul

cmake -B build\ReleaseOV -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_OPENVINO=ON \
    -DCMAKE_TOOLCHAIN_FILE="%VCPKG_DIR%\scripts\buildsystems\vcpkg.cmake"

if errorlevel 1 (
    echo If you continue to face CMAKE errors, make sure to install:
    echo   winget install Microsoft.VisualStudio.2022.BuildTools
    echo   Then run the "Developer Command Prompt for VS 2022" and launch this script from there.
    exit /b 1
)

cmake --build build\ReleaseOV --config Release
if errorlevel 1 exit /b 1

echo ============================================

echo Build completed successfully!
echo ============================================
echo Binaries: %CD%\build\ReleaseOV\bin
echo.

echo NOTE: To run, source setupvars.bat and pick a device:
echo   call "C:\Intel\openvino\setupvars.bat"
echo   set GGML_OPENVINO_DEVICE=CPU   ^&^& REM or GPU / NPU
echo   build\ReleaseOV\bin\llama-cli.exe -m model.gguf
echo.

endlocal
```
```