### Run llama-cpp-python API Server with Metal GPU

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Starts the llama-cpp-python API server, utilizing a specified GGUF model and enabling Metal GPU acceleration by setting the number of GPU layers. Omitting `--n_gpu_layers` will default to CPU usage.

```bash
# config your ggml model path
# make sure it is gguf v2
# make sure it is q4_0
export MODEL=[path to your llama.cpp ggml models]]/[ggml-model-name]]Q4_0.gguf
python3 -m llama_cpp.server --model $MODEL  --n_gpu_layers 1
```

--------------------------------

### Check Xcode Installation on macOS

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Verifies if Xcode command-line tools are installed and shows their path. If not installed, it provides the command to install them, which can be a lengthy process.

```bash
# check the path of your xcode install 
xcode-select -p

# xcode installed returns
# /Applications/Xcode-beta.app/Contents/Developer

# if xcode is missing then install it... it takes ages;
xcode-select --install
```

--------------------------------

### Install llama-cpp-python with Server Support

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Installs the llama-cpp-python library with the necessary dependencies for running the web server. This is the initial step before starting the server.

```bash
pip install llama-cpp-python[server]
```

--------------------------------

### Install Miniforge for macOS (ARM64)

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Downloads and installs the Miniforge distribution for macOS on ARM64 architecture. This is a prerequisite for managing conda environments.

```bash
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh
```

--------------------------------

### Install llama-cpp-python with Metal GPU Support

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Installs the latest version of llama-cpp-python, ensuring Metal GPU support is enabled via CMAKE_ARGS. It also installs the server component and uninstalls any previous versions.

```bash
pip uninstall llama-cpp-python -y
CMAKE_ARGS="-DGGML_METAL=on" pip install -U llama-cpp-python --no-cache-dir
pip install 'llama-cpp-python[server]'

# you should now have llama-cpp-python v0.1.62 or higher installed
llama-cpp-python 			 0.1.68
```

--------------------------------

### Get Server Help and Options

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Displays all available command-line options and arguments for the llama-cpp-python server. This is useful for understanding the full range of configurable settings.

```bash
python3 -m llama_cpp.server --help
```

--------------------------------

### Install Server with GPU (cuBLAS) Support

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Details the installation process for `llama-cpp-python` server with GPU (cuBLAS) support, including setting `CMAKE_ARGS` and `FORCE_CMAKE`. It also shows how to launch the server with GPU layer offloading.

```bash
CMAKE_ARGS="-DGGML_CUDA=on" FORCE_CMAKE=1 pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35
```

--------------------------------

### Install llama-cpp-python with pre-built CPU wheel

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs a pre-built wheel for llama-cpp-python with basic CPU support. This is an alternative to building from source.

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

```

--------------------------------

### Specify llama.cpp build configuration in requirements.txt

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Specifies the installation of llama-cpp-python with custom CMake arguments within a requirements.txt file. This example enables OpenBLAS support.

```text
llama-cpp-python -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

```

--------------------------------

### Install and Run OpenAI Compatible Web Server

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Provides commands to install the `llama-cpp-python` server package and run the web server. The server aims to be a drop-in replacement for the OpenAI API, allowing the use of llama.cpp compatible models with OpenAI clients.

```bash
pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf
```

--------------------------------

### Run llama.cpp Server with Config File (Python)

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

This command demonstrates how to start the llama.cpp Python server using a specified configuration file. The `--config_file` parameter points to the JSON configuration file, which can contain various server and model settings.

```bash
python3 -m llama_cpp.server --config_file <config_file>
```

--------------------------------

### Configure llama.cpp build with CMAKE_ARGS (Linux/Mac)

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python while configuring the llama.cpp build using the CMAKE_ARGS environment variable. This example enables OpenBLAS support.

```bash
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" \
  pip install llama-cpp-python

```

--------------------------------

### Example llama.cpp Server Configuration (JSON)

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

An example JSON configuration file for the llama.cpp Python server. This file defines server-level settings like host and port, and a list of models, each with its own configuration including model path, alias, chat format, and hardware acceleration settings.

```json
{
    "host": "0.0.0.0",
    "port": 8080,
    "models": [
        {
            "model": "models/OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
            "model_alias": "gpt-3.5-turbo",
            "chat_format": "chatml",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
            "model_alias": "gpt-4",
            "chat_format": "chatml",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/ggml_llava-v1.5-7b/ggml-model-q4_k.gguf",
            "model_alias": "gpt-4-vision-preview",
            "chat_format": "llava-1.5",
            "clip_model_path": "models/ggml_llava-v1.5-7b/mmproj-model-f16.gguf",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/mistral-7b-v0.1-GGUF/ggml-model-Q4_K.gguf",
            "model_alias": "text-davinci-003",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/replit-code-v1_5-3b-GGUF/replit-code-v1_5-3b.Q4_0.gguf",
            "model_alias": "copilot-codex",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 1024,
            "n_ctx": 9216
        }
    ]
}
```

--------------------------------

### Run the llama-cpp-python Server

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Starts the OpenAI compatible web server. Requires specifying the path to the model file. This command serves local models for client connections.

```bash
python3 -m llama_cpp.server --model <model_path>
```

--------------------------------

### Configure llama.cpp build with CMAKE_ARGS (Windows)

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python on Windows, configuring the llama.cpp build using the CMAKE_ARGS environment variable. This example enables OpenBLAS support.

```powershell
$env:CMAKE_ARGS = "-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install llama-cpp-python

```

--------------------------------

### Run Server for Multimodal Models

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Starts the server configured for multimodal models (like llava1.5). It requires specifying the model path, the clip model path for image embedding, and the appropriate chat format.

```bash
python3 -m llama_cpp.server --model <model_path> --clip_model_path <clip_model_path> --chat_format llava-1-5
```

--------------------------------

### Create and Activate Conda Environment

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Creates a new conda environment named 'llama' with Python 3.9.16 and then activates it. This isolates project dependencies.

```bash
conda create -n llama python=3.9.16
conda activate llama
```

--------------------------------

### Configure Server for Code Completion

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Starts the server with an increased context size (`n_ctx`) suitable for GitHub Copilot requests. This configuration is essential for enabling code completion functionality.

```bash
python3 -m llama_cpp.server --model <model_path> --n_ctx 16192
```

--------------------------------

### Configure llama.cpp build with pip config-settings

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs or upgrades llama-cpp-python using pip's --config-settings flag to pass CMake arguments. This example enables OpenBLAS support.

```bash
pip install --upgrade pip # ensure pip is up to date
pip install llama-cpp-python \
  -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

```

--------------------------------

### Install llama-cpp-python in Editable Mode (Shell)

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Clones the llama-cpp-python repository and installs it in editable mode. This method is recommended for development as it allows for direct code changes. It also covers installing with optional server dependencies or all optional dependencies.

```shell
git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python

# Upgrade pip (required for editable mode)
pip install --upgrade pip

# Install with pip
pip install -e .

# if you want to use the fastapi / openapi server
pip install -e '.[server]'

# to install all optional dependencies
pip install -e '.[all]'

# to clear the local build cache
make clean
```

--------------------------------

### Run Server for Function Calling

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Starts the server with specific chat format settings (`functionary-v2`) and the path to the Hugging Face tokenizer for models like 'functionary'. This enables structured function calling.

```bash
python3 -m llama_cpp.server --model <model_path_to_functionary_v2_model> --chat_format functionary-v2 --hf_pretrained_model_name_or_path <model_path_to_functionary_v2_tokenizer>
```

--------------------------------

### Install llama-cpp-python with OpenBLAS support

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python with OpenBLAS acceleration enabled. This requires setting specific CMAKE_ARGS environment variables before installation.

```bash
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python

```

--------------------------------

### Install llama-cpp-python with Vulkan support

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python with Vulkan support for GPU acceleration. This requires setting the GGML_VULKAN=on environment variable before installation.

```bash
CMAKE_ARGS="-DGGML_VULKAN=on" pip install llama-cpp-python

```

--------------------------------

### Install llama-cpp-python with pip

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs the llama-cpp-python package using pip. This command also builds the llama.cpp library from source.

```bash
pip install llama-cpp-python

```

--------------------------------

### Install llama-cpp-python with RPC support

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python with RPC support enabled. This requires sourcing the oneAPI environment script and setting the GGML_RPC=on environment variable.

```bash
source /opt/intel/oneapi/setvars.sh   
CMAKE_ARGS="-DGGML_RPC=on" pip install llama-cpp-python

```

--------------------------------

### Install llama-cpp-python with SYCL support

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python with SYCL support, typically for Intel GPUs. This requires sourcing the oneAPI environment script and setting specific CMAKE_ARGS.

```bash
source /opt/intel/oneapi/setvars.sh   
CMAKE_ARGS="-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx" pip install llama-cpp-python

```

--------------------------------

### Download Llama Models from Hugging Face Hub

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

This Python snippet demonstrates how to download Llama models in GGUF format directly from Hugging Face Hub using the `from_pretrained` method. Ensure the `huggingface-hub` package is installed.

```python
llm = Llama.from_pretrained(
    repo_id="Qwen/Qwen2-0.5B-Instruct-GGUF",
    filename="*q8_0.gguf",
    verbose=False
)
```

--------------------------------

### Install llama-cpp-python with pre-built CUDA wheel

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs a pre-built wheel for llama-cpp-python with CUDA support for a specific CUDA version. Replace '<cuda-version>' with the appropriate tag (e.g., cu121).

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/<cuda-version>

```

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

```

--------------------------------

### Install llama-cpp-python with pre-built Metal wheel

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs a pre-built wheel for llama-cpp-python with Metal (MPS) support. This option is available for compatible macOS and Python versions.

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

```

--------------------------------

### Tokenize Prompt using Low-level API

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

An example using the low-level `ctypes` binding to the `llama.cpp` C API to tokenize a given prompt. It includes initialization, model loading, context creation, tokenization, and context freeing.

```python
import llama_cpp
import ctypes
llama_cpp.llama_backend_init(False) # Must be called once at the start of each program
params = llama_cpp.llama_context_default_params()
# use bytes for char * params
model = llama_cpp.llama_load_model_from_file(b"./models/7b/llama-model.gguf", params)
ctx = llama_cpp.llama_new_context_with_model(model, params)
max_tokens = params.n_ctx
# use ctypes arrays for array params
tokens = (llama_cpp.llama_token * int(max_tokens))()
n_tokens = llama_cpp.llama_tokenize(ctx, b"Q: Name the planets in the solar system? A: ", tokens, max_tokens, llama_cpp.c_bool(True))llama_cpp.llama_free(ctx)
```

--------------------------------

### Install llama-cpp-python with Metal GPU on M1 Mac

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

This command installs or upgrades llama-cpp-python on an M1 Mac, enabling Metal GPU acceleration. It specifies the target architecture and processor for optimal performance on Apple Silicon.

```bash
CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DGGML_METAL=on" pip install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python
```

--------------------------------

### Run Tests for llama-cpp-python (Shell)

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Executes the test suite for the llama-cpp-python package using pytest. This command should be run after installing the package in editable mode to ensure all components are functioning correctly.

```shell
pytest
```

--------------------------------

### Load Functionary v2 Model with HF Tokenizer in Python

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

This example demonstrates how to load a Functionary v2 model using `Llama.from_pretrained` in llama-cpp-python. It specifies the model repository and filename, sets the chat format to 'functionary-v2', and crucially, provides an `LlamaHFTokenizer` instance loaded from the same model repository to ensure compatibility.

```python
from llama_cpp import Llama
from llama_cpp.llama_tokenizer import LlamaHFTokenizer
llm = Llama.from_pretrained(
  repo_id="meetkai/functionary-small-v2.2-GGUF",
  filename="functionary-small-v2.2.q4_0.gguf",
  chat_format="functionary-v2",
  tokenizer=LlamaHFTokenizer.from_pretrained("meetkai/functionary-small-v2.2-GGUF")
)

```

--------------------------------

### Install llama-cpp-python with CUDA support

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python with CUDA acceleration enabled. This requires setting the GGML_CUDA=on environment variable before installation.

```bash
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

```

--------------------------------

### Server Settings Reference

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Reference for server settings that can be configured via command-line arguments, environment variables, or the config file.

```APIDOC
## Server Options Reference

### `host`
- **Description**: Listen address for the server.
- **Type**: string
- **Default**: "localhost"

### `port`
- **Description**: Listen port for the server.
- **Type**: integer
- **Default**: 8000

### `ssl_keyfile`
- **Description**: Path to the SSL key file for HTTPS.
- **Type**: string | null
- **Default**: null

### `ssl_certfile`
- **Description**: Path to the SSL certificate file for HTTPS.
- **Type**: string | null
- **Default**: null

### `api_key`
- **Description**: API key for authentication. If set, all requests require authentication.
- **Type**: string | null
- **Default**: null

### `interrupt_requests`
- **Description**: Whether to interrupt requests when a new request is received.
- **Type**: boolean
- **Default**: true

### `disable_ping_events`
- **Description**: Disable EventSource pings (may be needed for some clients).
- **Type**: boolean
- **Default**: false

### `root_path`
- **Description**: The root path for the server. Useful when running behind a reverse proxy.
- **Type**: string
- **Default**: ""

### `models` (within config file)
- **Description**: List of model configurations to load.
- **Type**: array of `ModelSettings`
- **Default**: []
```

--------------------------------

### Configure Web Server Host, Port, and Prompt Format

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Illustrates how to configure the `llama-cpp-python` web server to bind to specific hosts and ports, and to set the prompt format, such as 'chatml', for better model compatibility.

```bash
python3 -m llama_cpp.server --host 0.0.0.0
python3 -m llama_cpp.server --port 8000
python3 -m llama_cpp.server --model models/7B/llama-model.gguf --chat_format chatml
```

--------------------------------

### Install llama-cpp-python with hipBLAS (ROCm) support

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python with hipBLAS acceleration for AMD GPUs using ROCm. This requires setting the GGML_HIPBLAS=on environment variable before installation.

```bash
CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python

```

--------------------------------

### Install llama-cpp-python with Metal (MPS) support

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Installs llama-cpp-python with Metal (MPS) acceleration enabled for Apple Silicon. This requires setting the GGML_METAL=on environment variable before installation.

```bash
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

```

--------------------------------

### Build and Test with Makefile (Shell)

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Utilizes the provided Makefile for common development workflows. 'make build' compiles the project, and 'make test' runs the test suite, offering a streamlined approach to development tasks.

```shell
make build
make test
```

--------------------------------

### Model Loading and Configuration

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

This section details the parameters used for loading and configuring the language model, including paths, GPU offloading, and context settings.

```APIDOC
## Model Loading and Configuration

### Description
Parameters for loading and configuring the language model, including paths, GPU offloading, and context settings.

### Method
N/A (Configuration parameters)

### Endpoint
N/A (Configuration parameters)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **model** (str) - Required - The path to the model to use for generating completions.
- **model_alias** (str) - Optional - The alias of the model to use for generating completions.
- **n_gpu_layers** (int) - Optional - The number of layers to put on the GPU. Set -1 to move all to GPU. Defaults to 0.
- **split_mode** (enum) - Optional - The split mode to use. Defaults to `llama_cpp.LLAMA_SPLIT_MODE_LAYER`.
- **main_gpu** (int) - Optional - Main GPU to use. Defaults to 0.
- **tensor_split** (list[float]) - Optional - Split layers across multiple GPUs in proportion.
- **vocab_only** (bool) - Optional - Whether to only return the vocabulary. Defaults to False.
- **use_mmap** (bool) - Optional - Use mmap. Defaults to `llama_cpp.llama_supports_mmap()`.
- **use_mlock** (bool) - Optional - Use mlock. Defaults to `llama_cpp.llama_supports_mlock()`.
- **kv_overrides** (list[str]) - Optional - List of model kv overrides in the format key=type:value where type is one of (bool, int, float).
- **rpc_servers** (str) - Optional - Comma-separated list of RPC servers for offloading.
- **seed** (int) - Optional - Random seed. -1 for random. Defaults to `llama_cpp.LLAMA_DEFAULT_SEED`.
- **n_ctx** (int) - Optional - The context size. Defaults to 2048.
- **n_batch** (int) - Optional - The batch size to use per eval. Defaults to 512.
- **n_ubatch** (int) - Optional - The physical batch size used by llama.cpp. Defaults to 512.
- **numa** (bool or int) - Optional - Enable NUMA support. Defaults to False.
- **chat_format** (str) - Optional - Chat format to use.
- **clip_model_path** (str) - Optional - Path to a CLIP model for multi-modal chat completion.
- **cache** (bool) - Optional - Use a cache to reduce processing times. Defaults to False.
- **cache_type** (Literal["ram", "disk"]) - Optional - The type of cache to use. Defaults to "ram".
- **cache_size** (int) - Optional - The size of the cache in bytes. Defaults to 2 << 30.
- **hf_tokenizer_config_path** (str) - Optional - Path to a HuggingFace tokenizer_config.json file.
- **hf_pretrained_model_name_or_path** (str) - Optional - Model name or path for HuggingFace tokenizer.
- **hf_model_repo_id** (str) - Optional - Model repo ID for HuggingFace tokenizer.
- **draft_model** (str) - Optional - Method for speculative decoding (e.g., "prompt-lookup-decoding").
- **draft_model_num_pred_tokens** (int) - Optional - Number of tokens to predict using the draft model. Defaults to 10.
- **type_k** (int) - Optional - Type of the key cache quantization.
- **type_v** (int) - Optional - Type of the value cache quantization.
- **verbose** (bool) - Optional - Whether to print debug information. Defaults to True.

### Request Example
```json
{
  "model": "./models/7B/ggml-model-q4_0.bin",
  "n_gpu_layers": 32,
  "n_ctx": 4096,
  "use_mlock": true
}
```

### Response
#### Success Response (200)
N/A (Configuration parameters)

#### Response Example
N/A (Configuration parameters)
```

--------------------------------

### Configure Windows Build with w64devkit

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

This snippet shows how to configure the build environment on Windows using w64devkit to specify the CMAKE generator and compiler paths. This is useful when encountering issues with finding 'nmake' or 'CMAKE_C_COMPILER'.

```powershell
$env:CMAKE_GENERATOR = "MinGW Makefiles"
$env:CMAKE_ARGS = "-DGGML_OPENBLAS=on -DCMAKE_C_COMPILER=C:/w64devkit/bin/gcc.exe -DCMAKE_CXX_COMPILER=C:/w64devkit/bin/g++.exe"
```

--------------------------------

### Run llama-cpp-python Server with Docker

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Shows the Docker command to run the `llama-cpp-python` server, mounting a local models directory and specifying the model to use via an environment variable.

```bash
docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/llama-model.gguf ghcr.io/abetlen/llama-cpp-python:latest
```

--------------------------------

### Server Configuration with JSON File

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

The server can be configured using a JSON file. This file supports all server and model options, and allows for specifying multiple models with aliases for routing requests.

```APIDOC
## POST /run

### Description
This endpoint allows for server configuration via a JSON config file.

### Method
POST

### Endpoint
`/run`

### Parameters
#### Query Parameters
- **config_file** (string) - Optional - Path to the JSON configuration file.

#### Request Body
- **models** (array) - Required - A list of model configurations.
  - **model** (string) - Required - Path to the model file.
  - **model_alias** (string) - Required - Alias for the model, used for routing requests.
  - **chat_format** (string) - Optional - The chat format to use for the model (e.g., 'chatml', 'llava-1.5').
  - **n_gpu_layers** (integer) - Optional - Number of layers to offload to the GPU.
  - **offload_kqv** (boolean) - Optional - Whether to offload K, Q, V matrices to the GPU.
  - **n_threads** (integer) - Optional - Number of threads to use for inference.
  - **n_batch** (integer) - Optional - Batch size for inference.
  - **n_ctx** (integer) - Optional - Context window size.
  - **clip_model_path** (string) - Optional - Path to the CLIP model for vision models.

### Request Example
```json
{
    "host": "0.0.0.0",
    "port": 8080,
    "models": [
        {
            "model": "models/OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
            "model_alias": "gpt-3.5-turbo",
            "chat_format": "chatml",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/ggml_llava-v1.5-7b/ggml-model-q4_k.gguf",
            "model_alias": "gpt-4-vision-preview",
            "chat_format": "llava-1.5",
            "clip_model_path": "models/ggml_llava-v1.5-7b/mmproj-model-f16.gguf",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        }
    ]
}
```

### Response
#### Success Response (200)
- **message** (string) - A confirmation message indicating successful configuration.

#### Response Example
```json
{
  "message": "Server configured successfully."
}
```
```

--------------------------------

### Interact with Multimodal Server using OpenAI Client

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Demonstrates how to use the OpenAI Python client to interact with the running multimodal server. It sends a request with both text and an image URL for processing.

```python
from openai import OpenAI

client = OpenAI(base_url="http://<host>:<port>/v1", api_key="sk-xxx")
response = client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "<image_url>"
                    },
                },
                {"type": "text", "text": "What does the image say"},
            ],
        }
    ],
)
print(response)
```

--------------------------------

### ServerSettings Class Definition (Python)

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

The `ServerSettings` class in `llama_cpp/server/settings.py` provides base settings for configuring the FastAPI and Uvicorn server. It includes options for host, port, SSL certificates, API key authentication, request interruption, and root path.

```python
class ServerSettings(BaseSettings):
    """Server settings used to configure the FastAPI and Uvicorn server."""

    # Uvicorn Settings
    host: str = Field(default="localhost", description="Listen address")
    port: int = Field(default=8000, description="Listen port")
    ssl_keyfile: Optional[str] = Field(
        default=None, description="SSL key file for HTTPS"
    )
    ssl_certfile: Optional[str] = Field(
        default=None, description="SSL certificate file for HTTPS"
    )
    # FastAPI Settings
    api_key: Optional[str] = Field(
        default=None,
        description="API key for authentication. If set all requests need to be authenticated.",
    )
    interrupt_requests: bool = Field(
        default=True,
        description="Whether to interrupt requests when a new request is received.",
    )
    disable_ping_events: bool = Field(
        default=False,
        description="Disable EventSource pings (may be needed for some clients).",
    )
    root_path: str = Field(
        default="",
        description="The root path for the server. Useful when running behind a reverse proxy.",
    )
```

--------------------------------

### Load Model from Hugging Face Hub via Web Server

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Demonstrates using the `--hf_model_repo_id` flag to load models directly from the Hugging Face Hub when running the `llama-cpp-python` web server.

```bash
python3 -m llama_cpp.server --hf_model_repo_id Qwen/Qwen2-0.5B-Instruct-GGUF --model '*q8_0.gguf'
```

--------------------------------

### Basic Text Completion with Llama High-level API

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

This Python snippet shows how to initialize the Llama class and perform basic text completion. It includes parameters for model path, GPU layers, seed, context window, max tokens, stop sequences, and echoing the prompt.

```python
from llama_cpp import Llama

llm = Llama(
      model_path="./models/7B/llama-model.gguf",
      # n_gpu_layers=-1, # Uncomment to use GPU acceleration
      # seed=1337, # Uncomment to set a specific seed
      # n_ctx=2048, # Uncomment to increase the context window
)
output = llm(
      "Q: Name the planets in the solar system? A: ", # Prompt
      max_tokens=32, # Generate up to 32 tokens, set to None to generate up to the end of the context window
      stop=["Q:", "\n"], # Stop generating just before the model would generate a new question
      echo=True # Echo the prompt back in the output
) # Generate a completion, can also call create_completion
print(output)
```

--------------------------------

### Context and LoRA Configuration in llama-cpp-python

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Manages the size of the last N tokens for repeat penalty calculations and specifies paths for LoRA base models or LoRA files to be applied.

```python
last_n_tokens_size = Field(default=64, ge=0, description='Last n tokens to keep for repeat penalty calculation.')
lora_base = Field(default=None, description='Optional path to base model, useful if using a quantized base model and you want to apply LoRA to an f16 model.')
lora_path = Field(default=None, description='Path to a LoRA file to apply to the model.')
```

--------------------------------

### Configure VS Code for Code Completion

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Sets up VS Code settings to use the local llama-cpp-python server for code completion. It overrides the default proxy URL to point to the running server.

```json
{
    // ...
    "github.copilot.advanced": {
        "debug.testOverrideProxyUrl": "http://<host>:<port>",
        "debug.overrideProxyUrl": "http://<host>:<port>"
    }
    // ...
}
```

--------------------------------

### Attention and Kernel Options in llama-cpp-python

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Enables experimental matrix multiplication kernels and Flash Attention for performance improvements. It also controls whether to return all logits and offload key-value queries to the GPU.

```python
mul_mat_q = Field(default=True, description='if true, use experimental mul_mat_q kernels')
logits_all = Field(default=True, description='Whether to return logits.')
offload_kqv = Field(default=True, description='Whether to offload kqv to the GPU.')
flash_attn = Field(default=False, description='Whether to use flash attention.')
```

--------------------------------

### OpenAI API v1 Compatibility

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Provides compatibility with the OpenAI API v1 specification, returning pydantic models instead of dictionaries for chat completions.

```APIDOC
## POST /v1/chat/completions

### Description
Initiates a chat completion request compatible with OpenAI API v1, returning pydantic models.

### Method
POST

### Endpoint
`/v1/chat/completions`

### Parameters
(Same as `/chat/completions` endpoint, but response types are pydantic models)

### Request Example
```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"}
  ]
}
```

### Response
#### Success Response (200)
- **choices** (list[ChatCompletionChoice]) - A list of completion choices.
  - **message** (ChatCompletionMessage) - The generated message.
    - **role** (str) - The role of the message sender ('assistant').
    - **content** (str) - The content of the message.

#### Response Example
(Returns pydantic models, structure similar to standard chat completion response)
```

--------------------------------

### Chat Completion with Llama-cpp-python

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Demonstrates how to initiate a chat completion using the `Llama` class. It requires a model path and optionally a chat format. The `create_chat_completion` method takes a list of messages with roles and content, and returns the model's response.

```python
from llama_cpp import Llama
llm = Llama(
      model_path="path/to/llama-2/llama-model.gguf",
      chat_format="llama-2"
)
llm.create_chat_completion(
      messages = [
          {"role": "system", "content": "You are an assistant who perfectly describes images."},
          {
              "role": "user",
              "content": "Describe this image in detail please."
          }
      ]
)

```

--------------------------------

### Multi-modal and Cache Configuration in llama-cpp-python

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Configures the path to a CLIP model for multi-modal chat and enables/configures caching to reduce processing times. Cache types and sizes can be specified.

```python
clip_model_path = Field(default=None, description='Path to a CLIP model to use for multi-modal chat completion.')
cache = Field(default=False, description='Use a cache to reduce processing times for evaluated prompts.')
cache_type = Field(default='ram', description='The type of cache to use. Only used if cache is True.')
cache_size = Field(default=(2 << 30), description='The size of the cache in bytes. Only used if cache is True.')
```

--------------------------------

### ConfigFileSettings Class Definition (Python)

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

The `ConfigFileSettings` class in `llama_cpp/server/settings.py` defines the structure for the JSON configuration file. It inherits from `ServerSettings` and includes a list of `ModelSettings` to configure multiple models.

```python
class ConfigFileSettings(ServerSettings):
    """Configuration file format settings."""

    models: List[ModelSettings] = Field(default=[], description="Model configs")
```

--------------------------------

### Define Llama Model Server Settings in Python

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

This Python code defines the `ModelSettings` class, inheriting from `BaseSettings`. It specifies various configuration options for loading and running a Llama model, including model path, GPU layer offloading, context size, batching, sampling parameters, and LoRA settings. These settings are crucial for customizing model behavior and performance.

```python
class ModelSettings(BaseSettings):
    """Model settings used to load a Llama model."""

    model: str = Field(
        description="The path to the model to use for generating completions."
    )
    model_alias: Optional[str] = Field(
        default=None,
        description="The alias of the model to use for generating completions.",
    )
    # Model Params
    n_gpu_layers: int = Field(
        default=0,
        ge=-1,
        description="The number of layers to put on the GPU. The rest will be on the CPU. Set -1 to move all to GPU.",
    )
    split_mode: int = Field(
        default=llama_cpp.LLAMA_SPLIT_MODE_LAYER,
        description="The split mode to use.",
    )
    main_gpu: int = Field(
        default=0,
        ge=0,
        description="Main GPU to use.",
    )
    tensor_split: Optional[List[float]] = Field(
        default=None,
        description="Split layers across multiple GPUs in proportion.",
    )
    vocab_only: bool = Field(
        default=False, description="Whether to only return the vocabulary."
    )
    use_mmap: bool = Field(
        default=llama_cpp.llama_supports_mmap(),
        description="Use mmap.",
    )
    use_mlock: bool = Field(
        default=llama_cpp.llama_supports_mlock(),
        description="Use mlock.",
    )
    kv_overrides: Optional[List[str]] = Field(
        default=None,
        description="List of model kv overrides in the format key=type:value where type is one of (bool, int, float). Valid true values are (true, TRUE, 1), otherwise false.",
    )
    rpc_servers: Optional[str] = Field(
        default=None,
        description="comma seperated list of rpc servers for offloading",
    )
    # Context Params
    seed: int = Field(
        default=llama_cpp.LLAMA_DEFAULT_SEED, description="Random seed. -1 for random."
    )
    n_ctx: int = Field(default=2048, ge=0, description="The context size.")
    n_batch: int = Field(
        default=512, ge=1, description="The batch size to use per eval."
    )
    n_ubatch: int = Field(
        default=512, ge=1, description="The physical batch size used by llama.cpp"
    )
    n_threads: int = Field(
        default=max(multiprocessing.cpu_count() // 2, 1),
        ge=1,
        description="The number of threads to use. Use -1 for max cpu threads",
    )
    n_threads_batch: int = Field(
        default=max(multiprocessing.cpu_count(), 1),
        ge=0,
        description="The number of threads to use when batch processing. Use -1 for max cpu threads",
    )
    rope_scaling_type: int = Field(
        default=llama_cpp.LLAMA_ROPE_SCALING_TYPE_UNSPECIFIED
    )
    rope_freq_base: float = Field(default=0.0, description="RoPE base frequency")
    rope_freq_scale: float = Field(
        default=0.0, description="RoPE frequency scaling factor"
    )
    yarn_ext_factor: float = Field(default=-1.0)
    yarn_attn_factor: float = Field(default=1.0)
    yarn_beta_fast: float = Field(default=32.0)
    yarn_beta_slow: float = Field(default=1.0)
    yarn_orig_ctx: int = Field(default=0)
    mul_mat_q: bool = Field(
        default=True, description="if true, use experimental mul_mat_q kernels"
    )
    logits_all: bool = Field(default=True, description="Whether to return logits.")
    embedding: bool = Field(default=False, description="Whether to use embeddings.")
    offload_kqv: bool = Field(
        default=True, description="Whether to offload kqv to the GPU."
    )
    flash_attn: bool = Field(
        default=False, description="Whether to use flash attention."
    )
    # Sampling Params
    last_n_tokens_size: int = Field(
        default=64,
        ge=0,
        description="Last n tokens to keep for repeat penalty calculation.",
    )
    # LoRA Params
    lora_base: Optional[str] = Field(
        default=None,
        description="Optional path to base model, useful if using a quantized base model and you want to apply LoRA to an f16 model.",
    )
    lora_path: Optional[str] = Field(
        default=None,
        description="Path to a LoRA file to apply to the model.",
    )
    # Backend Params

```

--------------------------------

### Load and Use Llava Multi-modal Model with llama-cpp-python

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

Demonstrates loading a multi-modal model like Llava1.5 using a custom chat handler and processing chat messages with images. Requires specifying the clip model path and increasing n_ctx. The input can include both text and image URLs.

```python
from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llava15ChatHandler
chat_handler = Llava15ChatHandler(clip_model_path="path/to/llava/mmproj.bin")
llm = Llama(
  model_path="./path/to/llava/llama-model.gguf",
  chat_handler=chat_handler,
  n_ctx=2048, # n_ctx should be increased to accommodate the image embedding
)
llm.create_chat_completion(
    messages = [
        {"role": "system", "content": "You are an assistant who perfectly describes images."},
        {
            "role": "user",
            "content": [
                {"type" : "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" } } 
            ]
        }
    ]
)

```

--------------------------------

### HuggingFace Tokenizer Configuration in llama-cpp-python

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Specifies paths and model identifiers for HuggingFace tokenizers. This allows integration with pretrained tokenizers from HuggingFace model repositories.

```python
hf_tokenizer_config_path = Field(default=None, description='The path to a HuggingFace tokenizer_config.json file.')
hf_pretrained_model_name_or_path = Field(default=None, description='The model name or path to a pretrained HuggingFace tokenizer model. Same as you would pass to AutoTokenizer.from_pretrained().')
hf_model_repo_id = Field(default=None, description='The model repo id to use for the HuggingFace tokenizer model.')
```

--------------------------------

### Server Configuration Attributes

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

This section outlines the configurable attributes for the Llama CPP Python server, including API key, request interruption behavior, ping event disabling, and root path configuration.

```APIDOC
## Server Configuration Attributes

### Description
Configuration options for the Llama CPP Python server.

### Method
N/A (Configuration Attributes)

### Endpoint
N/A (Configuration Attributes)

### Parameters
#### Class Attributes
- **api_key** (string) - Optional - API key for authentication. If set, all requests need to be authenticated.
- **interrupt_requests** (boolean) - Optional - Whether to interrupt requests when a new request is received. Defaults to True.
- **disable_ping_events** (boolean) - Optional - Disable EventSource pings. May be needed for some clients. Defaults to False.
- **root_path** (string) - Optional - The root path for the server. Useful when running behind a reverse proxy. Defaults to an empty string.

### Request Example
```json
{
  "api_key": "your_secret_api_key",
  "interrupt_requests": false,
  "disable_ping_events": true,
  "root_path": "/api/v1"
}
```

### Response
#### Success Response (200)
N/A (Configuration Attributes)

#### Response Example
N/A (Configuration Attributes)
```

--------------------------------

### OpenAI Compatible Text Completion Output Format

Source: https://llama-cpp-python.readthedocs.io/en/stable/index

This JSON structure represents the default output format for text completions generated by llama-cpp-python, which is compatible with the OpenAI API format. It includes completion details, model information, and token usage.

```json
{
  "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1679561337,
  "model": "./models/7B/llama-model.gguf",
  "choices": [
    {
      "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
      "index": 0,
      "logprobs": None,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 28,
    "total_tokens": 42
  }
}
```