### Complete VirtGPU Configuration Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/VirtGPU/configuration.md

Example environment variables for configuring VirtGPU on macOS with the Metal backend. Ensure the specified library paths and registration functions are correct for your setup.

```bash
# Hypervisor environment
export VIRGL_APIR_BACKEND_LIBRARY="/opt/llama.cpp/lib/libggml-virtgpu-backend.dylib"

# Backend configuration
export APIR_LLAMA_CPP_GGML_LIBRARY_PATH="/opt/llama.cpp/lib/libggml-metal.dylib"
export APIR_LLAMA_CPP_GGML_LIBRARY_REG="ggml_backend_metal_reg"

# Optional logging
export VIRGL_APIR_LOG_TO_FILE="/tmp/apir.log"
export APIR_LLAMA_CPP_LOG_TO_FILE="/tmp/ggml.log"

# Guest configuration
export GGML_REMOTING_USE_APIR_CAPSET=1
```

--------------------------------

### Setup Python Virtual Environment

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/examples/model-conversion/README.md

Sets up a Python virtual environment and installs dependencies. Ensure you are using Python 3.11.

```console
python3.11 -m venv venv
source venv/bin/activate
(venv) $ pip install -r requirements.txt
```

--------------------------------

### Configure Web UI Preferences via Command Line

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README.md

Example command to start llama-server with custom web UI preferences, such as disabling pasting long text as attachments and enabling Markdown rendering.

```bash
./llama-server -m model.gguf --webui-config '{"pasteLongTextToFileLen": 0, "renderUserContentAsMarkdown": true}'
```

--------------------------------

### Run Speculative Decoding Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/examples/speculative-simple/README.md

Execute the speculative decoding example with specified model paths, input file, and decoding parameters. Adjust model paths and parameters as needed for your setup.

```bash
./bin/llama-speculative-simple \
    -m  ../models/qwen2.5-32b-coder-instruct/ggml-model-q8_0.gguf \
    -md ../models/qwen2.5-1.5b-coder-instruct/ggml-model-q4_0.gguf \
    -f test.txt -c 0 -ngl 99 --color \
    --sampling-seq k --top-k 1 -fa --temp 0.0 \
    -ngld 99 --draft-max 16 --draft-min 5 --draft-p-min 0.9
```

--------------------------------

### Install Dependencies

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/webui/README.md

Navigate to the webui directory and install project dependencies using npm.

```bash
cd tools/server/webui
npm install
```

--------------------------------

### Install Conversion Script

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/CMakeLists.txt

Installs the 'convert_hf_to_gguf.py' script to the binary installation directory with execute permissions.

```cmake
install(
    FILES convert_hf_to_gguf.py
    PERMISSIONS
        OWNER_READ
        OWNER_WRITE
        OWNER_EXECUTE
        GROUP_READ
        GROUP_EXECUTE
        WORLD_READ
        WORLD_EXECUTE
    DESTINATION ${CMAKE_INSTALL_BINDIR})
```

--------------------------------

### Install Pkg-config File

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/CMakeLists.txt

Configures and installs the 'llama.pc' file for pkg-config.

```cmake
configure_file(cmake/llama.pc.in
        "${CMAKE_CURRENT_BINARY_DIR}/llama.pc"
        @ONLY)

install(FILES "${CMAKE_CURRENT_BINARY_DIR}/llama.pc"
        DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
```

--------------------------------

### Install Executable

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/examples/gguf-hash/CMakeLists.txt

Installs the built executable to the runtime path.

```cmake
install(TARGETS ${TARGET} RUNTIME)
```

--------------------------------

### Install Dependencies

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/tests/README.md

Installs necessary dependencies for running server tests. Ensure requirements.txt is present.

```shell
pip install -r requirements.txt
```

--------------------------------

### Install OpenCL using vcpkg (Windows)

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/OPENVINO.md

Clones the vcpkg repository, bootstraps it, installs OpenCL, and integrates vcpkg with Visual Studio.

```powershell
# Windows PowerShell
cd C:\
git clone https://github.com/microsoft/vcpkg
cd vcpkg
.\bootstrap-vcpkg.bat
.\vcpkg install opencl
# Optional but recommended: Integrate vcpkg with Visual Studio / CMake:
.\vcpkg integrate install
```

--------------------------------

### LLaDA Architecture CLI Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/examples/diffusion/README.md

Example for the LLaDA architecture using block-based scheduling for diffusion.

```bash
llama-diffusion-cli -m llada-8b.gguf -p "write code to train MNIST in pytorch" -ub 512 --diffusion-block-length 32 --diffusion-steps 256 --diffusion-visual
```

--------------------------------

### Install OpenVINO Runtime from Archive (Linux)

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/OPENVINO.md

Downloads and executes a script to install OpenVINO Runtime from an archive file on Linux. Verifies installation by checking the OpenVINO_DIR environment variable.

```bash
wget https://raw.githubusercontent.com/ravi9/misc-scripts/main/openvino/ov-archive-install/install-openvino-from-archive.sh
    chmod +x install-openvino-from-archive.sh
    ./install-openvino-from-archive.sh
```

```bash
echo $OpenVINO_DIR
```

--------------------------------

### Run Lookahead Decoding Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/examples/lookahead/README.md

This command demonstrates how to run the lookahead decoding example. It specifies the model, prompt, and various generation parameters.

```bash
llama-lookahead -hf ggml-org/Qwen2.5-Coder-3B-Q8_0-GGUF -p "// network server implemented in C\n// author: Peter Hacker\n\n#include" -e -ngl 99 -t 4 -n 512 -c 4096 -kvu
```

--------------------------------

### Install Build Tools on Windows

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/OPENVINO.md

Installs Git, Wget, and Ninja build tools using winget on Windows.

```powershell
# Windows PowerShell
winget install Git.Git
winget install GNU.Wget
winget install Ninja-build.Ninja
```

--------------------------------

### Install Library and Headers

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/CMakeLists.txt

Installs the main llama library, its public headers, and optionally the llama-common library.

```cmake
set_target_properties(llama
    PROPERTIES
        PUBLIC_HEADER "${LLAMA_PUBLIC_HEADERS}")

install(TARGETS llama LIBRARY PUBLIC_HEADER)

if (LLAMA_BUILD_COMMON)
    install(TARGETS llama-common LIBRARY)
endif()
```

--------------------------------

### Create Installable Package

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/snapdragon/linux.md

Generate an installable package of the built llama.cpp and zip it for deployment. This is done after the build process.

```bash
[d]/workspace> cmake --install build-snapdragon --prefix pkg-snapdragon
[d]/workspace> zip -r pkg-snapdragon.zip pkg-snapdragon
```

--------------------------------

### Run Interactive Chat Example (Bash)

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README.md

Command to run the interactive chat example using bash. Requires bash, curl, and jq.

```sh
bash chat.sh
```

--------------------------------

### Example: Use ngram-simple Speculative Decoding

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/speculative.md

This example demonstrates how to enable the 'ngram-simple' speculative decoding type when running the llama-server.

```bash
./llama-server [...] --spec-type ngram-simple
```

--------------------------------

### RND1 Architecture CLI Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/examples/diffusion/README.md

Example for the RND1 architecture using entropy-based selection, timestep scheduling, and temperature sampling.

```bash
llama-diffusion-cli -m RND1-Base-0910.gguf -p "write code to train MNIST in pytorch" -ub 512 --diffusion-algorithm 1 --diffusion-steps 256 --diffusion-visual --temp 0.5 --diffusion-eps 0.001
```

--------------------------------

### Dream Architecture CLI Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/examples/diffusion/README.md

Example for the Dream architecture using timestep-based scheduling and random selection for diffusion algorithm.

```bash
llama-diffusion-cli -m dream7b.gguf -p "write code to train MNIST in pytorch" -ub 512 --diffusion-eps 0.001 --diffusion-algorithm 3 --diffusion-steps 256 --diffusion-visual
```

--------------------------------

### Setup Build Environment and Compile

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/development/debugging-tests.md

Configure the build with debug settings and compile the test binaries. Adapt arguments as needed.

```bash
cmake -DCMAKE_BUILD_TYPE=Debug -DLLAMA_CUDA=1 -DLLAMA_FATAL_WARNINGS=ON ..
make -j
```

--------------------------------

### GET /models Response Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README.md

Example response for listing available models. The 'in_cache' field indicates if a model is stored locally, and the 'status' object details its current state.

```json
{
  "data": [{
    "id": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
    "in_cache": true,
    "path": "/Users/REDACTED/Library/Caches/llama.cpp/ggml-org_gemma-3-4b-it-GGUF_gemma-3-4b-it-Q4_K_M.gguf",
    "status": {
      "value": "loaded",
      "args": ["llama-server", "-ctx", "4096"]
    },
    ...
  }]
}
```

--------------------------------

### Launch Container Environment

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/VirtGPU/development.md

Set the container provider to libkrun and start the podman machine to prepare the containerized testing environment.

```bash
# Set container provider to libkrun
export CONTAINERS_MACHINE_PROVIDER=libkrun
podman machine start
```

--------------------------------

### Build Production Bundle

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README-dev.md

Run this command to create the production-ready bundle for the UI. Ensure `public/index.html` is generated before this step.

```bash
npm run build
```

--------------------------------

### Get Server Global Properties

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README.md

Use this GET endpoint to retrieve the current global properties of the server. To enable POST requests for modifying properties, start the server with the `--props` flag.

```json
{
  "default_generation_settings": {
    "id": 0,
    "id_task": -1,
    "n_ctx": 1024,
    "speculative": false,
    "is_processing": false,
    "params": {
      "n_predict": -1,
      "seed": 4294967295,
      "temperature": 0.800000011920929,
      "dynatemp_range": 0.0,
      "dynatemp_exponent": 1.0,
      "top_k": 40,
      "top_p": 0.949999988079071,
      "min_p": 0.05000000074505806,
      "xtc_probability": 0.0,
      "xtc_threshold": 0.10000000149011612,
      "typical_p": 1.0,
      "repeat_last_n": 64,
      "repeat_penalty": 1.0,
      "presence_penalty": 0.0,
      "frequency_penalty": 0.0,
      "dry_multiplier": 0.0,
      "dry_base": 1.75,
      "dry_allowed_length": 2,
      "dry_penalty_last_n": -1,
      "dry_sequence_breakers": [
        "\n",
        ":",
        "\"",
        "*"
      ],
      "mirostat": 0,
      "mirostat_tau": 5.0,
      "mirostat_eta": 0.10000000149011612,
      "stop": [],
      "max_tokens": -1,
      "n_keep": 0,
      "n_discard": 0,
      "ignore_eos": false,
      "stream": true,
      "n_probs": 0,
      "min_keep": 0,
      "grammar": "",
      "samplers": [
        "dry",
        "top_k",
        "typ_p",
        "top_p",
        "min_p",
        "xtc",
        "temperature"
      ],
      "speculative.n_max": 16,
      "speculative.n_min": 5,
      "speculative.p_min": 0.8999999761581421,
      "timings_per_token": false
    },
    "prompt": "",
    "next_token": {
      "has_next_token": true,
      "has_new_line": false,
      "n_remain": -1,
      "n_decoded": 0,
      "stopping_word": ""
    }
  },
  "total_slots": 1,
  "model_path": "../models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
  "chat_template": "...",
  "chat_template_caps": {},
  "modalities": {
    "vision": false
  },
  "media_marker": "<__media_YoNhud46VdDqbuFmKYEO9PY7A4ARzRfg__>",
  "build_info": "b(build number)-(build commit hash)",
  "is_sleeping": false
}
```

--------------------------------

### Example llama-cli Command with Grammar

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/grammars/README.md

This command demonstrates how to use a GBNF grammar file with the llama-cli tool for guided text generation.

```bash
./llama-cli -m <model> --grammar-file grammars/some-grammar.gbnf -p 'Some prompt'
```

--------------------------------

### Install and Set Up Environment Variables on Target Device

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/snapdragon/linux.md

Transfer the built package to the target Linux device, unzip it, and set the necessary environment variables for library paths.

```bash
$ unzip pkg-snapdragon.zip
$ cd pkg-snapdragon
$ export LD_LIBRARY_PATH=./lib
$ export ADSP_LIBRARY_PATH=./lib
```

--------------------------------

### Setup Python Environment for TTS Client

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/tts/README.md

Create a virtual environment and install Python dependencies (requests, numpy) required for the TTS client script.

```console
$ python3 -m venv venv
$ source venv/bin/activate
(venv) pip install requests numpy
```

--------------------------------

### Install k6 and SSE Extension

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/bench/README.md

Installs the k6 load testing tool and builds it with the xk6-sse extension for Server-Sent Events support. Requires Go version 1.21 or later.

```shell
go install go.k6.io/xk6/cmd/xk6@latest
$GOPATH/bin/xk6 build master \
--with github.com/phymbert/xk6-sse
```

--------------------------------

### Run Completion with Multiple Hexagon NPUs on Snapdragon

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/snapdragon/README.md

Execute a summary request using two Hexagon NPU sessions (HTP0, HTP1) for a large OLMoE-1B-7B model. This example shows how to specify multiple devices using the `D` variable and `NDEV=2`, and includes logs detailing the setup and memory usage for each NPU.

```bash
~/src/llama.cpp$ M=OLMoE-1B-7B-0125-Instruct-Q4_0.gguf NDEV=2 D=HTP0,HTP1 ./scripts/snapdragon/adb/run-completion.sh -f surfing.txt
... 
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v81
ggml-hex: allocating new session: HTP0
ggml-hex: allocating new session: HTP1
... 
load_tensors: offloading output layer to GPU
load_tensors: offloaded 17/17 layers to GPU
load_tensors:          CPU model buffer size =   143.86 MiB
load_tensors:         HTP1 model buffer size =     0.23 MiB
load_tensors:  HTP1-REPACK model buffer size =  1575.00 MiB
load_tensors:         HTP0 model buffer size =     0.28 MiB
load_tensors:  HTP0-REPACK model buffer size =  2025.00 MiB
... 
llama_context:        CPU  output buffer size =     0.19 MiB
llama_kv_cache:       HTP1 KV buffer size =   238.00 MiB
llama_kv_cache:       HTP0 KV buffer size =   306.00 MiB
llama_kv_cache: size =  544.00 MiB (  8192 cells,  16 layers,  1/1 seqs), K (q8_0):  272.00 MiB, V (q8_0):  272.00 MiB
llama_context:       HTP0 compute buffer size =    15.00 MiB
llama_context:       HTP1 compute buffer size =    15.00 MiB
llama_context:        CPU compute buffer size =    24.56 MiB
... 
llama_perf_context_print: prompt eval time =    1730.57 ms /   212 tokens (    8.16 ms per token,   122.50 tokens per second)
llama_perf_context_print:        eval time =    5624.75 ms /   257 runs   (   21.89 ms per token,    45.69 tokens per second)
llama_perf_context_print:       total time =    7377.33 ms /   469 tokens
llama_perf_context_print:    graphs reused =        255
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - HTP0 (Hexagon)     |  2048 = 2048 + (   0 =     0 +       0 +       0) +           0 |
llama_memory_breakdown_print: |   - HTP1 (Hexagon)     |  2048 = 2048 + (   0 =     0 +       0 +       0) +           0 |
llama_memory_breakdown_print: |   - Host               |                  742 =   144 +     544 +      54                |
llama_memory_breakdown_print: |   - HTP1-REPACK        |                 1575 =  1575 +       0 +       0                |
llama_memory_breakdown_print: |   - HTP0-REPACK        |                 2025 =  2025 +       0 +       0                |
```

--------------------------------

### Docker Compose Example for Llama.cpp Server

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README.md

This example demonstrates how to configure and run the llama.cpp server using docker compose, including setting model paths, context size, parallel processing, and port mapping. It shows alternative methods for model loading.

```yaml
services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      # alternatively, you can use "LLAMA_ARG_MODEL_URL" to download the model
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_ENDPOINT_METRICS: 1
      LLAMA_ARG_PORT: 8080
```

--------------------------------

### Web UI: Install Dependencies

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README-dev.md

Command to install Node.js dependencies for the Web UI. Ensure Node.js is installed before running.

```sh
cd tools/server/webui
npm i
```

--------------------------------

### Start Server with Legacy Completion UI

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README.md

Command to start the llama-server with the legacy completion-based web UI, using the --path argument to specify the directory.

```sh
./llama-server -m my_model.gguf -c 8192 --path ./tools/server/public_legacy
```

--------------------------------

### Install gguf Python Package

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/gguf-py/README.md

Install the gguf package using pip. For the visual editor, install with the 'gui' extra.

```sh
pip install gguf
```

```sh
pip install gguf[gui]
```

--------------------------------

### Verify GPU Driver Installation with clinfo

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/SYCL.md

Install and use 'clinfo' to verify that your GPU drivers are correctly installed and recognized by the system.

```shell
sudo apt install clinfo
sudo clinfo -l
```

--------------------------------

### Basic Multimodal CLI and Server Usage

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/multimodal.md

Demonstrates simple command-line usage for both the `llama-mtmd-cli` and `llama-server` tools with Hugging Face models. Also shows how to specify local model and mmproj files, and disable GPU offloading.

```sh
# simple usage with CLI
llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF
```

```sh
# simple usage with server
llama-server -hf ggml-org/gemma-3-4b-it-GGUF
```

```sh
# using local file
llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf
```

```sh
# no GPU offload
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --no-mmproj-offload
```

--------------------------------

### Install OpenCL Headers and Library for Windows Arm64

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/OPENCL.md

Installs OpenCL headers and the ICD loader using CMake and Ninja, specifying installation paths.

```powershell
mkdir -p ~/dev/llm

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers
mkdir build && cd build
cmake .. -G Ninja \
  -DBUILD_TESTING=OFF \
  -DOPENCL_HEADERS_BUILD_TESTING=OFF \
  -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF \
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader
mkdir build && cd build
cmake .. -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install
```

--------------------------------

### Verify CUDA Installation

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/CUDA-FEDORA.md

Checks the installed version of the NVIDIA CUDA Compiler (nvcc) to confirm that CUDA is correctly installed and accessible in the system's PATH.

```bash
nvcc --version
```

--------------------------------

### Run LLaVA 1.6 CLI

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/multimodal/llava.md

Example command to run the multimodal CLI with the converted LLaVA 1.6 model. Note the different mmproj path.

```console
./llama-mtmd-cli -m ../llava-v1.6-vicuna-7b/ggml-model-f16.gguf --mmproj vit/mmproj-model-f16.gguf
```

--------------------------------

### Install BLIS

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/BLIS.md

Install the compiled BLIS library to the system.

```bash
sudo make install
```

--------------------------------

### Install llama.cpp via Nix (Non-flake)

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/install.md

Install llama.cpp using Nix for non-flake enabled systems. This command installs the package using the specified Nixpkgs channel.

```sh
nix-env --file '<nixpkgs>' --install --attr llama-cpp
```

--------------------------------

### JSON_NATIVE Tool Call Examples

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/autoparser.md

Demonstrates standard OpenAI-style, Mistral Nemo with array wrapper, and Apertus-style JSON tool calls.

```json
<tool_call>
{"name": "get_weather", "arguments": {"location": "Paris", "unit": "celsius"}}
</tool_call>
```

```json
[TOOL_CALLS]
[{"name": "calculate", "arguments": {"expr": "2+2"}}]
```

```json
{"get_weather": {"location": "Paris"}}
```

--------------------------------

### Configure MSVC for Examples

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/ggml/CMakeLists.txt

Applies MSVC-specific build settings to targets related to ggml examples, provided GGML_BUILD_EXAMPLES is enabled. This ensures examples compile correctly on Windows.

```cmake
if (GGML_BUILD_EXAMPLES)
    configure_msvc_target(common-ggml)
    configure_msvc_target(common)

    configure_msvc_target(mnist-common)
    configure_msvc_target(mnist-eval)
    configure_msvc_target(mnist-train)

    configure_msvc_target(gpt-2-ctx)
    configure_msvc_target(gpt-2-alloc)
    configure_msvc_target(gpt-2-backend)
    configure_msvc_target(gpt-2-sched)
    configure_msvc_target(gpt-2-quantize)
    configure_msvc_target(gpt-2-batched)

    configure_msvc_target(gpt-j)
    configure_msvc_target(gpt-j-quantize)

    configure_msvc_target(magika)
    configure_msvc_target(yolov3-tiny)
    configure_msvc_target(sam)

    configure_msvc_target(simple-ctx)
    configure_msvc_target(simple-backend)
endif()
```

--------------------------------

### Build and Install llama.cpp for Android

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/android.md

After configuration, build the project in release mode and then install it to a specified directory. Adjust the -j{n} and {install-dir} parameters as needed.

```bash
cmake --build build-android --config Release -j{n}
```

```bash
cmake --install build-android --prefix {install-dir} --config Release
```

--------------------------------

### Initialize Llama.cpp Turboquant App

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/public/index.html

This script initializes the SvelteKit base path and starts the application by dynamically importing './bundle.js' and calling its start method with the current script's parent element. Ensure this script is placed correctly in your HTML to bootstrap the application.

```javascript
{
  __sveltekit__ = {
    base: new URL('.', location).pathname.slice(0, -1)
  };
  const element = document.currentScript.parentElement;
  import("./bundle.js").then((app) => {
    app.start(element)
  });
}
```

--------------------------------

### Completion Endpoint Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README.md

Example of how to use the completion endpoint with curl.

```APIDOC
## POST /completion

### Description
This endpoint is used for text completion.

### Method
POST

### Endpoint
/completion

### Request Body
- prompt (string) - Required - The prompt to send to the model.
- n_predict (integer) - Optional - The number of tokens to predict.

### Request Example
```sh
curl --request POST \
    --url http://localhost:8080/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'
```
```

--------------------------------

### Run llama.cpp Retrieval Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/examples/retrieval/README.md

Execute the retrieval example with specified model, top-k value, context files, chunk size, and chunk separator. This command embeds the provided files and initiates an interactive query loop.

```bash
llama-retrieval --model ./models/bge-base-en-v1.5-f16.gguf --top-k 3 --context-file README.md --context-file License --chunk-size 100 --chunk-separator .
```

--------------------------------

### Install OpenCL on Linux

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/OPENVINO.md

Installs OpenCL development libraries and headers on Linux systems.

```bash
sudo apt install ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd
```

--------------------------------

### Basic text completion with llama-simple

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/README.md

A minimal example demonstrating basic text completion using llama.cpp. This is useful for developers integrating llama.cpp into their applications.

```bash
llama-simple -m model.gguf

# Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of
```

--------------------------------

### Enable oneAPI Environment

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/SYCL.md

Source this script to set up the oneAPI running environment. This is necessary before using SYCL devices.

```sh
source /opt/intel/oneapi/setvars.sh
```

--------------------------------

### Install llama-gguf-split Executable

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/gguf-split/CMakeLists.txt

Conditionally installs the 'llama-gguf-split' executable at runtime if the LLAMA_TOOLS_INSTALL option is enabled.

```cmake
if(LLAMA_TOOLS_INSTALL)
    install(TARGETS ${TARGET} RUNTIME)
endif()
```

--------------------------------

### Build and Test llama-cpp-turboquant

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/rocm-mi300x-test-results.md

Instructions for cloning the repository, checking out the correct branch, and building the project for both MI300X and MI355X GPUs. Includes a command to run the benchmark test.

```bash
git clone https://github.com/TheTom/llama-cpp-turboquant.git
cd llama-cpp-turboquant
git checkout feature/turboquant-kv-cache
```

```bash
# MI300X (gfx942) — works without code changes
cmake -B build -DGGML_HIP=ON -DCMAKE_BUILD_TYPE=Release -DAMDGPU_TARGETS="gfx942"
cmake --build build --config Release -j
```

```bash
# MI355X (gfx950) — requires CDNA4 define patch (see commit)
cmake -B build -DGGML_HIP=ON -DCMAKE_BUILD_TYPE=Release -DAMDGPU_TARGETS="gfx950"
cmake --build build --config Release -j
```

```bash
# Test
HIP_VISIBLE_DEVICES=0 ./build/bin/llama-bench \
  -m model.gguf -ctk turbo3 -ctv turbo3 -ngl 99 -r 3 -p 512 -n 128
```

--------------------------------

### Install CUDA Meta-Package

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/CUDA-FEDORA.md

Installs the CUDA toolkit and all associated packages required for CUDA development.

```bash
sudo dnf install cuda
```

--------------------------------

### Install Nvidia Driver Libraries (Guest)

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/CUDA-FEDORA.md

Installs the necessary NVIDIA driver libraries and related packages within the toolbox if they are not supplied by the host system.

```bash
sudo dnf install nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced
```

--------------------------------

### Install RHEL/CentOS Dependencies

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/CANN.md

Installs essential build tools and kernel headers for RHEL/CentOS systems.

```bash
sudo yum makecache
sudo yum install -y gcc python3 python3-pip kernel-headers-$(uname -r) kernel-devel-$(uname -r)
```

--------------------------------

### Install Ubuntu/Debian Dependencies

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/CANN.md

Installs essential build tools and kernel headers for Ubuntu/Debian systems.

```bash
sudo apt-get update
sudo apt-get install -y gcc python3 python3-pip linux-headers-$(uname -r)
```

--------------------------------

### Run llama-tts Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/tts/README.md

Execute the TTS example directly. Requires llama.cpp to be built with SSL support for automatic model download. Plays the generated audio using 'aplay'.

```console
$ build/bin/llama-tts --tts-oute-default -p "Hello world" && aplay output.wav
```

--------------------------------

### Build and Run Server Tests

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/tests/README.md

Builds the llama-server binary and then executes the test suite. Assumes the project root is two directories above the tests directory.

```shell
cd ../../..
cmake -B build
cmake --build build --target llama-server
```

```shell
./tests.sh
```

--------------------------------

### Preprocessor Directive Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/CONTRIBUTING.md

Example of a preprocessor directive with a corresponding closing comment. This is a placeholder for future guidelines.

```cpp
#ifdef FOO
#endif // FOO
```

--------------------------------

### Install Vulkan Dependencies on Debian/Ubuntu

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/build.md

Install Vulkan development libraries and tools on Debian-based systems using apt-get.

```sh
sudo apt-get install libvulkan-dev glslc spirv-headers
```

--------------------------------

### Install requirements for model conversion

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/build-s390x.md

Install the necessary Python packages before converting models. Ensure you have a requirements.txt file.

```bash
pip3 install -r requirements.txt
```

--------------------------------

### Install zDNN Library

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/zDNN.md

Clone the zDNN repository, configure the installation path, and build/install the library. It is preferred to compile from source rather than using package managers.

```sh
git clone --recurse-submodules https://github.com/IBM/zDNN
cd zDNN

autoreconf .
./configure --prefix=/opt/zdnn-libs

make build
sudo make install
```

--------------------------------

### Start Llama.cpp Server with ZenDNN

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/backend/ZenDNN.md

Run the Llama.cpp server with ZenDNN acceleration, specifying the model and network configuration.

```sh
# Set optimal configuration
export ZENDNNL_MATMUL_ALGO=1    # Blocked AOCL DLP algo for best performance

# Start server
./build/bin/llama-server \
    -m models/Llama-3.1-8B-Instruct.BF16.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -t 64
```

--------------------------------

### Define Installation Directories

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/CMakeLists.txt

Sets cache variables for installation directories, including include, library, and binary paths.

```cmake
include(GNUInstallDirs)

set(LLAMA_INCLUDE_INSTALL_DIR ${CMAKE_INSTALL_INCLUDEDIR} CACHE PATH "Location of header  files")
set(LLAMA_LIB_INSTALL_DIR     ${CMAKE_INSTALL_LIBDIR}     CACHE PATH "Location of library files")
set(LLAMA_BIN_INSTALL_DIR     ${CMAKE_INSTALL_BINDIR}     CACHE PATH "Location of binary  files")
```

--------------------------------

### Start Server with Native Support Models

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/docs/function-calling.md

Use these commands to start the llama-server with models that have native support for tool-aware Jinja templates. Ensure the model name and GGUF type are correct.

```shell
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
```

```shell
llama-server --jinja -fa -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K_L
```

```shell
llama-server --jinja -fa -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M
```

--------------------------------

### Download Model (PHI-2 Example)

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/bench/README.md

Downloads a specific model file (PHI-2 in q4_0 quantization) using a helper script. Adjust the repository and file path as needed for other models.

```shell
../../../scripts/hf.sh --repo ggml-org/models --file phi-2/ggml-model-q4_0.gguf
```

--------------------------------

### POST /models/unload Payload Example

Source: https://github.com/thetom/llama-cpp-turboquant/blob/feature/turboquant-kv-cache/tools/server/README.md

Example payload for the POST /models/unload endpoint, specifying the model name to be unloaded.

```json
{
  "model": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
}
```