llama-cpp-python

https://llama-cpp-python.readthedocs.io/en/stable
Admin
Python bindings for llama.cpp that provide low-level C API access and high-level APIs for text...
Tokens:17,368
Snippets:155
Trust Score:8.3
Update:4 weeks ago
Show doc for...
Tokens:
Raw
### Install and run the server

Source: https://llama-cpp-python.readthedocs.io/en/stable

Install the server package and start the server with a local model file.

```bash
pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf

```

--------------------------------

### Install CUDA 12.1 wheel example

Source: https://llama-cpp-python.readthedocs.io/en/stable

Example of installing the CUDA 12.1 pre-built wheel using the cu121 index URL.

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

```

--------------------------------

### Install with GPU (cuBLAS) support and run

Source: https://llama-cpp-python.readthedocs.io/en/stable

Install the server with CUDA support and run with GPU layers.

```bash
CMAKE_ARGS="-DGGML_CUDA=on" FORCE_CMAKE=1 pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35

```

--------------------------------

### Install the server with pip

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Install the llama-cpp-python server package using pip. Run this command in your terminal to install the server component.

```bash
pip install llama-cpp-python[server]

```

--------------------------------

### Tokenize prompt using low-level ctypes API

Source: https://llama-cpp-python.readthedocs.io/en/stable

This example demonstrates initializing the backend, loading a model, creating a context, tokenizing a prompt, and freeing resources. It uses bytes for char* parameters and ctypes arrays for array parameters. Must call llama_backend_init() once at program start.

```python
import llama_cpp
import ctypes
llama_cpp.llama_backend_init()  # Must be called once at the start of each program
model_params = llama_cpp.llama_model_default_params()
ctx_params = llama_cpp.llama_context_default_params()
prompt = b"Q: Name the planets in the solar system? A: "
# use bytes for char * params
model = llama_cpp.llama_model_load_from_file(b"./models/7b/llama-model.gguf", model_params)
ctx = llama_cpp.llama_init_from_model(model, ctx_params)
vocab = llama_cpp.llama_model_get_vocab(model)
max_tokens = ctx_params.n_ctx
# use ctypes arrays for array params
tokens = (llama_cpp.llama_token * int(max_tokens))()
n_tokens = llama_cpp.llama_tokenize(vocab, prompt, len(prompt), tokens, max_tokens, True, False)
llama_cpp.llama_free(ctx)
llama_cpp.llama_model_free(model)
```

--------------------------------

### Install pre-built wheel with CPU support

Source: https://llama-cpp-python.readthedocs.io/en/stable

Installs a pre-built wheel with basic CPU support using the extra-index-url flag to access the pre-built wheel repository.

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

```

--------------------------------

### Install with Vulkan backend

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set GGML_VULKAN=on environment variable before installing to enable Vulkan support.

```bash
CMAKE_ARGS="-DGGML_VULKAN=on" pip install llama-cpp-python

```

--------------------------------

### Install with RPC backend

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set GGML_RPC=on environment variable before installing to enable RPC support. Requires sourcing the Intel oneAPI environment first.

```bash
source /opt/intel/oneapi/setvars.sh   
CMAKE_ARGS="-DGGML_RPC=on" pip install llama-cpp-python

```

--------------------------------

### Install Vulkan pre-built wheel

Source: https://llama-cpp-python.readthedocs.io/en/stable

Install a pre-built Vulkan wheel for Linux or Windows using the vulkan index URL.

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/vulkan

```

--------------------------------

### Install with CUDA backend

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set GGML_CUDA=on environment variable before installing to enable CUDA support.

```bash
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

```

--------------------------------

### Check and install Xcode command line tools

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Checks the Xcode installation path and installs the command line tools if missing. Required before building llama-cpp-python from source.

```bash
# check the path of your xcode install 
xcode-select -p

# xcode installed returns
# /Applications/Xcode-beta.app/Contents/Developer

# if xcode is missing then install it... it takes ages;
xcode-select --install

```

--------------------------------

### Install llama-cpp-python with Metal support

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Uninstalls any existing llama-cpp-python, then installs the latest version with Metal GPU support using CMAKE_ARGS. Also installs the server extras. Requires Xcode to be installed for compilation.

```bash
pip uninstall llama-cpp-python -y
CMAKE_ARGS="-DGGML_METAL=on" pip install -U llama-cpp-python --no-cache-dir
pip install 'llama-cpp-python[server]'

# you should now have llama-cpp-python v0.1.62 or higher installed
llama-cpp-python         0.1.68

```

--------------------------------

### Install with SYCL backend

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set GGML_SYCL=on and compiler environment variables before installing to enable SYCL support. Requires sourcing the Intel oneAPI environment first.

```bash
source /opt/intel/oneapi/setvars.sh   
CMAKE_ARGS="-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx" pip install llama-cpp-python

```

--------------------------------

### Install with Metal backend

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set GGML_METAL=on environment variable before installing to enable Metal (MPS) support.

```bash
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

```

--------------------------------

### Start the server with chat-template kwargs

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Start the server with chat-template kwargs passed from the CLI. Use --chat_format to specify the format and --chat_template_kwargs to pass JSON-encoded kwargs.

```bash
python3 -m llama_cpp.server \
  --model <model_path> \
  --chat_format chatml \
  --chat_template_kwargs '{"enable_thinking": true}'

```

--------------------------------

### Install HIP Radeon pre-built wheel (Windows)

Source: https://llama-cpp-python.readthedocs.io/en/stable

Install a pre-built HIP Radeon wheel for Windows using the hip-radeon index URL.

```powershell
pip install llama-cpp-python `
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/hip-radeon

```

--------------------------------

### Start server with config file using --config_file

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Command to launch the server with a configuration file. The config file can be specified via the --config_file parameter or the CONFIG_FILE environment variable.

```bash
python3 -m llama_cpp.server --config_file <config_file>

```

--------------------------------

### Install with OpenBLAS backend

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set GGML_BLAS and GGML_BLAS_VENDOR environment variables before installing to enable OpenBLAS CPU support.

```bash
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python

```

--------------------------------

### Start the server with a model

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Start the OpenAI-compatible server with a specified model path. Replace <model_path> with the actual path to your GGUF model file.

```bash
python3 -m llama_cpp.server --model <model_path>

```

--------------------------------

### Install Miniforge for MacOS (ARM64)

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Downloads and runs the Miniforge installer for Apple Silicon Macs to set up a conda environment.

```bash
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh

```

--------------------------------

### Install llama-cpp-python with pip

Source: https://llama-cpp-python.readthedocs.io/en/stable

Installs the llama-cpp-python package, which builds llama.cpp from source and installs it alongside the Python package. If the build fails, add --verbose to see the full cmake build log.

```bash
pip install llama-cpp-python

```

--------------------------------

### Install with HIP/ROCm backend

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set GGML_HIP=on environment variable before installing to enable HIP/ROCm support for AMD cards.

```bash
CMAKE_ARGS="-DGGML_HIP=on" pip install llama-cpp-python

```

--------------------------------

### Install Metal pre-built wheel

Source: https://llama-cpp-python.readthedocs.io/en/stable

Install a pre-built Metal wheel for macOS 11.0+ with Python 3.10-3.12.

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

```

--------------------------------

### Run server for multimodal llava model

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Start the server with a multimodal model, specifying the clip model path and the llava-1-5 chat format. Requires a GGUF model and a clip model.

```bash
python3 -m llama_cpp.server --model <model_path> --clip_model_path <clip_model_path> --chat_format llava-1-5

```

--------------------------------

### Install ROCm pre-built wheel (Linux)

Source: https://llama-cpp-python.readthedocs.io/en/stable

Install a pre-built ROCm wheel for Linux using the rocm72 index URL.

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/rocm72

```

--------------------------------

### Install CUDA pre-built wheel

Source: https://llama-cpp-python.readthedocs.io/en/stable

Install a pre-built CUDA wheel by specifying the CUDA version in the extra index URL. Replace <cuda-version> with cu118, cu121, cu122, cu123, cu124, cu125, cu130, or cu132.

```bash
pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/<cuda-version>

```

--------------------------------

### Clone and install llama-cpp-python in editable mode

Source: https://llama-cpp-python.readthedocs.io/en/stable

Clones the repository with submodules and installs the package in editable mode. Includes optional extras for development, server, and all dependencies. Requires pip upgrade for editable mode. Run 'make clean' to clear the local build cache.

```bash
git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python

# Upgrade pip (required for editable mode)
pip install --upgrade pip

# Install with pip
pip install -e .

# install development tooling (tests, docs, ruff)
pip install -e '.[dev]'

# if you want to use the fastapi / openapi server
pip install -e '.[server]'

# to install all optional dependencies
pip install -e '.[all]'

# to clear the local build cache
make clean
```

--------------------------------

### Run server for function calling with functionary-v2

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Start the server with functionary-v2 chat format and specify the tokenizer path. Requires a functionary model in GGUF format and the corresponding HF tokenizer.

```bash
python3 -m llama_cpp.server --model <model_path_to_functionary_v2_model> --chat_format functionary-v2 --hf_pretrained_model_name_or_path <model_path_to_functionary_v2_tokenizer>

```

--------------------------------

### Install with CMake args via CLI

Source: https://llama-cpp-python.readthedocs.io/en/stable

Installs the package using the -C / --config-settings flag to pass CMake arguments directly through the pip command. This ensures pip is up to date first.

```bash
pip install --upgrade pip # ensure pip is up to date
pip install llama-cpp-python \
  -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

```

--------------------------------

### Run llama-cpp-python API server with Metal GPU

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Sets the MODEL environment variable to a GGUF v2 Q4_0 model path and starts the server with GPU layers enabled. Omitting --n_gpu_layers 1 will use CPU instead.

```bash
# config your ggml model path
# make sure it is gguf v2
# make sure it is q4_0
export MODEL=[path to your llama.cpp ggml models]]/[ggml-model-name]]Q4_0.gguf
python3 -m llama_cpp.server --model $MODEL  --n_gpu_layers 1

```

--------------------------------

### Specify CMake args in requirements.txt

Source: https://llama-cpp-python.readthedocs.io/en/stable

Shows how to specify CMake arguments in a requirements.txt file using the -C flag, allowing the same configuration to be reused across installations.

```text
# requirements.txt

llama-cpp-python -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

```

--------------------------------

### Set CMAKE_ARGS for Windows

Source: https://llama-cpp-python.readthedocs.io/en/stable

Sets the CMAKE_ARGS environment variable to enable OpenBLAS acceleration on Windows, then installs the package.

```powershell
# Windows
$env:CMAKE_ARGS = "-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install llama-cpp-python

```

--------------------------------

### Multi-model config with aliases and GPU settings

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Example of a multi-model configuration with three models, each with distinct settings like model_alias, n_gpu_layers, offload_kqv, n_threads, n_batch, and n_ctx. The server routes requests based on the 'model' parameter matching model_alias.

```json
{
    "host": "0.0.0.0",
    "port": 8080,
    "models": [
        {
            "model": "models/OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
            "model_alias": "gpt-3.5-turbo",
            "chat_format": "chatml",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
            "model_alias": "gpt-4",
            "chat_format": "chatml",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/ggml_llava-v1.5-7b/ggml-model-q4_k.gguf",
            "model_alias": "gpt-4-vision-preview",
            "chat_format": "llava-1-5",
            "clip_model_path": "models/ggml_llava-v1.5-7b/mmproj-model-f16.gguf",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/mistral-7b-v0.1-GGUF/ggml-model-Q4_K.gguf",
            "model_alias": "text-davinci-003",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 512,
            "n_ctx": 2048
        },
        {
            "model": "models/replit-code-v1_5-3b-GGUF/replit-code-v1_5-3b.Q4_0.gguf",
            "model_alias": "copilot-codex",
            "n_gpu_layers": -1,
            "offload_kqv": true,
            "n_threads": 12,
            "n_batch": 1024,
            "n_ctx": 9216
        }
    ]
}

```

--------------------------------

### Load multi-modal model from Hugging Face Hub with MoondreamChatHandler

Source: https://llama-cpp-python.readthedocs.io/en/stable

Use MoondreamChatHandler.from_pretrained to load a multi-modal model from the Hugging Face Hub. The example shows how to load the model and generate a chat completion with an image URL.

```python
from llama_cpp import Llama
from llama_cpp.llama_chat_format import MoondreamChatHandler

chat_handler = MoondreamChatHandler.from_pretrained(
  repo_id="vikhyatk/moondream2",
  filename="*mmproj*",
)

llm = Llama.from_pretrained(
  repo_id="vikhyatk/moondream2",
  filename="*text-model*",
  chat_handler=chat_handler,
  n_ctx=2048, # n_ctx should be increased to accommodate the image embedding
)

response = llm.create_chat_completion(
    messages = [
        {
            "role": "user",
            "content": [
                {"type" : "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" } }
            ]
        }
    ]
)
print(response["choices"][0]["text"])

```

--------------------------------

### Set CMAKE_ARGS for Linux and Mac

Source: https://llama-cpp-python.readthedocs.io/en/stable

Sets the CMAKE_ARGS environment variable to enable OpenBLAS acceleration on Linux and Mac, then installs the package.

```bash
# Linux and Mac
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" \
  pip install llama-cpp-python

```

--------------------------------

### Initialize Llama with chatml-function-calling and create chat completion with tools

Source: https://llama-cpp-python.readthedocs.io/en/stable

Use this snippet to set up a Llama instance with the 'chatml-function-calling' chat format and call create_chat_completion with a tools list and tool_choice to extract structured data. The example extracts a user's name and age from a message. Ensure the model path points to a compatible GGUF file.

```python
from llama_cpp import Llama
llm = Llama(model_path="path/to/chatml/llama-model.gguf", chat_format="chatml-function-calling")
llm.create_chat_completion(
      messages = [
        {
          "role": "system",
          "content": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. The assistant calls functions with appropriate input when necessary"

        },
        {
          "role": "user",
          "content": "Extract Jason is 25 years old"
        }
      ],
      tools=[{
        "type": "function",
        "function": {
          "name": "UserDetail",
          "parameters": {
            "type": "object",
            "title": "UserDetail",
            "properties": {
              "name": {
                "title": "Name",
                "type": "string"
              },
              "age": {
                "title": "Age",
                "type": "integer"
              }
            },
            "required": [ "name", "age" ]
          }
        }
      }],
      tool_choice={
        "type": "function",
        "function": {
          "name": "UserDetail"
        }
      }
)

```

--------------------------------

### Single-model config with chat_template_kwargs

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Example of a single-model configuration where chat_template_kwargs is set directly on the model entry. This allows passing model-specific parameters like enable_thinking.

```json
{
    "models": [
        {
            "model": "models/Qwen3.5-0.8B/qwen3.5-0.8b-q8_0.gguf",
            "chat_format": "chatml",
            "chat_template_kwargs": {
                "enable_thinking": true
            }
        }
    ]
}

```

--------------------------------

### Load multi-modal model with Llava15ChatHandler

Source: https://llama-cpp-python.readthedocs.io/en/stable

Use Llava15ChatHandler to load a multi-modal model (e.g., llava) with a clip model. Increase n_ctx to accommodate image embeddings. The example shows how to pass an image URL in a chat completion request.

```python
from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llava15ChatHandler

chat_handler = Llava15ChatHandler(clip_model_path="path/to/llava/mmproj.bin")
llm = Llama(
  model_path="./path/to/llava/llama-model.gguf",
  chat_handler=chat_handler,
  n_ctx=2048, # n_ctx should be increased to accommodate the image embedding
)

llm.create_chat_completion(
    messages = [
        {"role": "system", "content": "You are an assistant who perfectly describes images."},
        {
            "role": "user",
            "content": [
                {"type" : "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" } }
            ]
        }
    ]
)

```

--------------------------------

### Convert local image to base64 data URI for multi-modal input

Source: https://llama-cpp-python.readthedocs.io/en/stable

Convert a local image file to a base64 data URI and use it in a chat completion message. The example shows how to construct the image_url content type with the data URI.

```python
import base64

def image_to_base64_data_uri(file_path):
    with open(file_path, "rb") as img_file:
        base64_data = base64.b64encode(img_file.read()).decode('utf-8')
        return f"data:image/png;base64,{base64_data}"

# Replace 'file_path.png' with the actual path to your PNG file
file_path = 'file_path.png'
data_uri = image_to_base64_data_uri(file_path)

messages = [
    {"role": "system", "content": "You are an assistant who perfectly describes images."},
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": data_uri }},
            {"type" : "text", "text": "Describe this image in detail please."}
        ]
    }
]

```

--------------------------------

### Show server help

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Display the full list of server options. Run this command to see all available CLI arguments and environment variables.

```bash
python3 -m llama_cpp.server --help

```

--------------------------------

### Typical build and test workflow with Makefile

Source: https://llama-cpp-python.readthedocs.io/en/stable

Uses the Makefile targets to build and test the package.

```bash
make build
make test
```

--------------------------------

### Set chat format to chatml

Source: https://llama-cpp-python.readthedocs.io/en/stable

Run the server with the chatml prompt format.

```bash
python3 -m llama_cpp.server --model models/7B/llama-model.gguf --chat_format chatml

```

--------------------------------

### Basic text completion with Llama class

Source: https://llama-cpp-python.readthedocs.io/en/stable

Shows how to instantiate Llama and generate a completion. The output is in OpenAI compatible format. The prompt is echoed back when echo=True. Stop tokens prevent generating new questions.

```python
from llama_cpp import Llama

llm = Llama(
      model_path="./models/7B/llama-model.gguf",
      # n_gpu_layers=-1, # Uncomment to use GPU acceleration
      # seed=1337, # Uncomment to set a specific seed
      # n_ctx=2048, # Uncomment to increase the context window
)
output = llm(
      "Q: Name the planets in the solar system? A: ", # Prompt
      max_tokens=32, # Generate up to 32 tokens, set to None to generate up to the end of the context window
      stop=["Q:", "\n"], # Stop generating just before the model would generate a new question
      echo=True # Echo the prompt back in the output
) # Generate a completion, can also call create_completion
print(output)
```

--------------------------------

### Download model from Hugging Face Hub with from_pretrained

Source: https://llama-cpp-python.readthedocs.io/en/stable

Downloads a GGUF model from Hugging Face using the from_pretrained method. Requires the huggingface-hub package. By default downloads to the Hugging Face cache directory.

```python
llm = Llama.from_pretrained(
    repo_id="lmstudio-community/Qwen3.5-0.8B-GGUF",
    filename="*Q8_0.gguf",
    verbose=False
)
```

--------------------------------

### Run server for code completion with increased context

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Run the server with a substantially increased context size (--n_ctx 16192) for GitHub Copilot code completion requests. Requires a code completion model in GGUF format.

```bash
python3 -m llama_cpp.server --model <model_path> --n_ctx 16192

```

--------------------------------

### Configure VS Code for GitHub Copilot proxy

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Update .vscode/settings.json to point GitHub Copilot to the local server. Replace <host> and <port> with your server's address.

```json
{
    // ...
    "github.copilot.advanced": {
        "debug.testOverrideProxyUrl": "http://<host>:<port>",
        "debug.overrideProxyUrl": "http://<host>:<port>"
    }
    // ...
}

```

--------------------------------

### Run server with Docker image from GHCR

Source: https://llama-cpp-python.readthedocs.io/en/stable

Use this command to run the server in a Docker container. It mounts a local models directory, sets the MODEL environment variable to the model file inside the container, and exposes port 8000. Requires Docker and a compatible model file.

```bash
docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/llama-model.gguf ghcr.io/abetlen/llama-cpp-python:latest
```

--------------------------------

### Check formatting and linting with ruff

Source: https://llama-cpp-python.readthedocs.io/en/stable

Checks code formatting and linting using ruff. Includes Makefile targets 'make lint' and 'make format' as alternatives.

```bash
python -m ruff check llama_cpp tests
python -m ruff format --check llama_cpp tests

# or use the Makefile targets
make lint
make format
```

--------------------------------

### Load model from Hugging Face Hub

Source: https://llama-cpp-python.readthedocs.io/en/stable

Use the --hf_model_repo_id flag to load a model from the Hugging Face Hub.

```bash
python3 -m llama_cpp.server --hf_model_repo_id lmstudio-community/Qwen3.5-0.8B-GGUF --model '*Q8_0.gguf'

```

--------------------------------

### Fix Apple Silicon architecture error

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set CMAKE_ARGS to build for arm64 with Metal support to fix the incompatible architecture error on M1 Macs.

```bash
CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DGGML_METAL=on" pip install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python

```

--------------------------------

### ServerSettings class definition

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Defines server settings for FastAPI and Uvicorn, including host, port, SSL, API key, request interruption, ping events, and root path.

```python
class ServerSettings(BaseSettings):
    """Server settings used to configure the FastAPI and Uvicorn server."""

    # Uvicorn Settings
    host: str = Field(default="localhost", description="Listen address")
    port: int = Field(default=8000, description="Listen port")
    ssl_keyfile: Optional[str] = Field(
        default=None, description="SSL key file for HTTPS"
    )
    ssl_certfile: Optional[str] = Field(
        default=None, description="SSL certificate file for HTTPS"
    )
    # FastAPI Settings
    api_key: Optional[str] = Field(
        default=None,
        description="API key for authentication. If set all requests need to be authenticated.",
    )
    interrupt_requests: bool = Field(
        default=True,
        description="Whether to interrupt requests when a new request is received.",
    )
    disable_ping_events: bool = Field(
        default=False,
        description="Disable EventSource pings (may be needed for some clients).",
    )
    root_path: str = Field(
        default="",
        description="The root path for the server. Useful when running behind a reverse proxy.",
    )

```

--------------------------------

### ConfigFileSettings class definition

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Defines the configuration file format settings, inheriting from ServerSettings. Includes a 'models' field that holds a list of ModelSettings.

```python
class ConfigFileSettings(ServerSettings):
    """Configuration file format settings."""

    models: List[ModelSettings] = Field(default=[], description="Model configs")

```

--------------------------------

### Adjust context window with n_ctx

Source: https://llama-cpp-python.readthedocs.io/en/stable

Set the n_ctx parameter to change the context window size. Default is 512 tokens.

```python
llm = Llama(model_path="./models/7B/llama-model.gguf", n_ctx=2048)

```

--------------------------------

### OpenAI compatible completion output format

Source: https://llama-cpp-python.readthedocs.io/en/stable

Default output format for completions generated by llama-cpp-python. Shows the structure with id, object, created, model, choices, and usage fields.

```json
{
  "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1679561337,
  "model": "./models/7B/llama-model.gguf",
  "choices": [
    {
      "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
      "index": 0,
      "logprobs": None,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 28,
    "total_tokens": 42
  }
}
```

--------------------------------

### Create and activate conda environment

Source: https://llama-cpp-python.readthedocs.io/en/stable/install/macos

Creates a new conda environment named 'llama' with Python 3.9.16 and activates it.

```bash
conda create -n llama python=3.9.16
conda activate llama

```

--------------------------------

### Use OpenAI client to send image and text

Source: https://llama-cpp-python.readthedocs.io/en/stable/server

Use the official OpenAI Python client to send a chat completion request with an image and text to the local server. The server must be running with a multimodal model.

```python
from openai import OpenAI

client = OpenAI(base_url="http://<host>:<port>/v1", api_key="sk-xxx")
response = client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "<image_url>"
                    },
                },
                {"type": "text", "text": "What does the image say"},
            ],
        }
    ],
)
print(response)

```

--------------------------------

### Chat completion with llama-2 chat format

Source: https://llama-cpp-python.readthedocs.io/en/stable

Uses create_chat_completion with a pre-registered chat format. The model formats messages into a single prompt. Set verbose=True to see the selected chat format.

```python
from llama_cpp import Llama
llm = Llama(
      model_path="path/to/llama-2/llama-model.gguf",
      chat_format="llama-2"
)
llm.create_chat_completion(
      messages = [
          {"role": "system", "content": "You are an assistant who perfectly describes images."},
          {
              "role": "user",
              "content": "Describe this image in detail please."
          }
      ]
)
```

--------------------------------

### Run tests with pytest

Source: https://llama-cpp-python.readthedocs.io/en/stable

Runs the test suite using pytest.

```bash
pytest
```
Tokens:
Raw