### Clone and Install PowerInfer

Source: https://github.com/tiiny-ai/powerinfer/blob/main/README.md

Initial setup commands to clone the repository and install required Python dependencies.

```bash
git clone https://github.com/Tiiny-AI/PowerInfer
cd PowerInfer
pip install -r requirements.txt # install Python helpers' dependencies
```

--------------------------------

### Install Node.js Dependencies and Run Web UI Dev Server

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/README.md

Commands to install dependencies and start the development server for the llama.cpp web UI. Requires Node.js to be installed.

```bash
# make sure you have nodejs installed
cd tools/server/webui
npm i

# to run the dev server
npm run dev
```

--------------------------------

### Install {fmt} via vcpkg

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/doc/get-started.md

Download and install {fmt} using the vcpkg package manager. This involves cloning the vcpkg repository, bootstrapping it, integrating it with your system, and then installing the fmt package.

```bash
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg install fmt
```

--------------------------------

### Run Interactive Chat Example (Bash)

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/README.md

Execute the provided chat example script using Bash. This requires bash, curl, and jq to be installed on your system.

```shell
bash chat.sh
```

--------------------------------

### Run llama.cpp Retrieval Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/examples/retrieval/README.md

Compile and run the retrieval example, specifying model, top-k value, context files, chunk size, and chunk separator. This process chunks and embeds files, then starts an interactive query loop.

```bash
make -j && ./llama-retrieval --model ./models/bge-base-en-v1.5-f16.gguf --top-k 3 --context-file README.md --context-file License --chunk-size 100 --chunk-separator .
```

--------------------------------

### Run llama.cpp Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/examples/simple-cmake-pkg/README.md

Execute the built llama.cpp example, specifying the model path and a prompt.

```sh
./build/llama-simple-cmake-pkg -m ./models/llama-7b-v2/ggml-model-f16.gguf "Hello my name is"
```

--------------------------------

### Run Interactive Chat Example (Node.js)

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/README.md

Execute the provided chat example script using Node.js. Ensure you have Node.js version 16 or later installed.

```shell
node chat.mjs
```

--------------------------------

### Install GoogleTest on Unix Systems

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/googletest/googletest/README.md

Build and install the compiled libraries to the system path.

```bash
make
sudo make install    # Install in /usr/local/ by default
```

--------------------------------

### Install build and twine

Source: https://github.com/tiiny-ai/powerinfer/blob/main/gguf-py/README.md

Install necessary tools for manual package distribution.

```sh
pip install build twine
```

--------------------------------

### TOML Configuration File Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/book/chapters/config.md

An example of a configuration file in TOML format, demonstrating comments and basic key-value pairs.

```toml
# Comments are supported, using a #
# The default section is [default], case-insensitive

value = 1
str = "A string"
vector = [1,2,3]

```

--------------------------------

### Run llama.cpp Simple Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/examples/simple/README.md

Execute the simple example with a model and prompt. Ensure the model path and prompt are correctly specified.

```bash
./simple ./models/llama-7b-v2/ggml-model-f16.gguf "Hello my name is"
```

--------------------------------

### Start llama-server for LLM model

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/tts/README.md

Starts a llama-server instance to serve the LLM model. This is used when running the TTS example with llama-server.

```console
$ ./build/bin/llama-server -m ./models/outetts-0.2-0.5B-q8_0.gguf --port 8020
```

--------------------------------

### Running the Example Program

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/book/chapters/flags.md

Demonstrates compiling and running the C++ flag example program, showing the help message and output with specific flag inputs.

```term
gitbook:examples $ g++ -std=c++11 flags.cpp
gitbook:examples $ ./a.out -h
Flag example program
Usage: ./a.out [OPTIONS]

Options:
  -h,--help                   Print this help message and exit
  -b,--bool                   This is a bool flag
  -i,--int                    This is an int flag
  -p,--plain                  This is a plain flag

gitbook:examples $ ./a.out -bii --plain -i
The flags program
Bool flag passed
Flag int: 3
Flag plain: 1
```

--------------------------------

### Install Dependencies

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/README.md

Install the necessary system packages and Python requirements.

```bash
sudo apt install clang-21 mold
```

```bash
pip install -r requirements.txt
```

--------------------------------

### Build and Install with Android NDK

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/android.md

Compile the project and install the binaries to a specified directory.

```bash
$ cmake --build build-android --config Release -j{n}
$ cmake --install build-android --prefix {install-dir} --config Release
```

--------------------------------

### Install Ascend Firmware

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/CANN.md

Install the required firmware package for the Ascend NPU.

```bash
# download driver from https://www.hiascend.com/hardware/firmware-drivers/community according to your system
# and install driver.
sudo sh Ascend-hdk-910b-npu-firmware_x.x.x.x.X.run --full
```

--------------------------------

### Start RPC server

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/rpc/README.md

Start the RPC server on a specific port.

```bash
$ bin/rpc-server -p 50052
create_backend: using CUDA backend
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA T1200 Laptop GPU, compute capability 7.5, VMM: yes
Starting RPC server on 0.0.0.0:50052
```

--------------------------------

### Verify Firmware Installation

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/CANN.md

Confirm successful firmware installation.

```bash
Firmware package installed successfully!
```

--------------------------------

### Install dependencies

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/tests/README.md

Install the required Python packages for the test suite.

```shell
pip install -r requirements.txt
```

--------------------------------

### Install {fmt} via Package Managers

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/doc-html/get-started.html

Commands to install the {fmt} library on different operating systems.

```bash
apt install libfmt-dev
```

```bash
brew install fmt
```

```bash
conda install -c conda-forge fmt
```

--------------------------------

### SYCL Device Output Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/SYCL.md

Example output showing detected SYCL devices and their properties.

```text
found 2 SYCL devices:
|  |                  |                                             |Compute   |Max compute|Max work|Max sub|               |
|ID|       Device Type|                                         Name|capability|units      |group   |group  |Global mem size|
|--|------------------|---------------------------------------------|----------|-----------|--------|-------|---------------|
| 0|[level_zero:gpu:0]|               Intel(R) Arc(TM) A770 Graphics|       1.3|        512|    1024|     32|    16225243136|
| 1|[level_zero:gpu:1]|                    Intel(R) UHD Graphics 770|       1.3|         32|     512|     32|    53651849216|
```

--------------------------------

### Install Vulkan SDK and Build on Linux

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/build.md

Commands to add the LunarG repository, install the SDK, and build the project for Linux environments.

```bash
wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | apt-key add -
wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list
apt update -y
apt-get install -y vulkan-sdk
# To verify the installation, use the command below:
vulkaninfo
```

```bash
cmake -B build -DGGML_VULKAN=1
cmake --build build --config Release
# Test the output binary (with "-ngl 33" to offload all layers to GPU)
./bin/llama-cli -m "PATH_TO_MODEL" -p "Hi you how are you" -n 50 -e -ngl 33 -t 4
```

--------------------------------

### Install Ascend Driver

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/CANN.md

Create the required user group and install the Ascend NPU driver.

```bash
# create driver running user.
sudo groupadd -g HwHiAiUser
sudo useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash
sudo usermod -aG HwHiAiUser $USER

# download driver from https://www.hiascend.com/hardware/firmware-drivers/community according to your system
# and install driver.
sudo sh Ascend-hdk-910b-npu-driver_x.x.x_linux-{arch}.run --full --install-for-all
```

--------------------------------

### Install {fmt} Subproject with Meson

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/doc-html/get-started.html

Install the {fmt} subproject from Meson WrapDB. Run this command from the root of your project.

```bash
meson wrap install fmt
```

--------------------------------

### INI Configuration File Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/README.md

An equivalent INI format example for CLI11 configuration. Supports comments and sections mapping to subcommands.

```ini
; Comments are supported, using a ;
; The default section is [default], case-insensitive

value = 1
str = "A string"
vector = 1 2 3
str_vector = "one" "two" "and three"

; Sections map to subcommands
[subcommand]
in_subcommand = Wow
sub.subcommand = true
```

--------------------------------

### Start the server

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/bench/README.md

Launches the llama-server with specific performance parameters for benchmarking.

```shell
llama-server --host localhost --port 8080 \
  --model ggml-model-q4_0.gguf \
  --cont-batching \
  --metrics \
  --parallel 8 \
  --batch-size 512 \
  --ctx-size 4096 \
  -ngl 33
```

--------------------------------

### Install Nvidia Driver Libraries (Guest)

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/CUDA-FEDORA.md

Installs the necessary NVIDIA driver libraries and related packages within the toolbox when the host system does not supply them.

```bash
sudo dnf install nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced
```

--------------------------------

### Install {fmt} via apt

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/doc/get-started.md

Install {fmt} on Debian, Ubuntu, or other Debian-based Linux distributions using the apt package manager.

```bash
apt install libfmt-dev
```

--------------------------------

### Create Project Directory

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/googletest/docs/quickstart-cmake.md

Initializes a new project directory and navigates into it. This is the first step in setting up a new project.

```bash
$ mkdir my_project && cd my_project
```

--------------------------------

### Generate Importance Matrix Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/imatrix/README.md

Example command to generate an importance matrix using the llama-imatrix tool with GPU offloading.

```bash
./llama-imatrix -m ggml-model-f16.gguf -f train-data.txt -ngl 99
```

--------------------------------

### Start llama-server for voice decoder model

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/tts/README.md

Starts a llama-server instance to serve the voice decoder model. This is used when running the TTS example with llama-server.

```console
./build/bin/llama-server -m ./models/wavtokenizer-large-75-f16.gguf --port 8021 --embeddings --pooling none
```

--------------------------------

### Install llama.cpp via Nix

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/install.md

Use these commands on Mac or Linux to install via Nix, choosing between flake-enabled or non-flake environments.

```sh
nix profile install nixpkgs#llama-cpp
```

```sh
nix-env --file '<nixpkgs>' --install --attr llama-cpp
```

--------------------------------

### TOML Configuration File Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/README.md

An example of a TOML file structure that can be parsed by CLI11. Comments, separators, and sections mapping to subcommands are supported.

```toml
# Comments are supported, using a #
# The default section is [default], case-insensitive

value = 1
value2 = 123_456 # a string with separators
str = "A string"
str2 = "A string\nwith new lines"
str3 = 'A literal "string"'
vector = [1,2,3]
str_vector = ["one","two","and three"]

# Sections map to subcommands
[subcommand]
in_subcommand = Wow
sub.subcommand = true
"sub"."subcommand2" = "string_value"
```

--------------------------------

### Quick Start - Infinite Text Generation

Source: https://github.com/tiiny-ai/powerinfer/blob/main/examples/main/README.md

Demonstrates how to generate continuous text from a prompt until stopped.

```APIDOC
## Infinite Text Generation Example

### Description
This example shows how to generate an "infinite" stream of text from a starting prompt. The generation can be stopped manually using Ctrl+C.

### Method
Command Line Execution

### Endpoint
N/A (Local execution)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
#### Unix-based systems (Linux, macOS, etc.):
```bash
./main -m models/7B/ggml-model.bin --ignore-eos -n -1 --random-prompt
```

#### Windows:
```powershell
main.exe -m models\7B\ggml-model.bin --ignore-eos -n -1 --random-prompt
```

### Response
#### Success Response (200)
Continuous text generation output.

#### Response Example
(Generated text stream)
```

--------------------------------

### Setup Python virtual environment for tts-outetts.py

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/tts/README.md

Creates a Python virtual environment and installs the 'requests' and 'numpy' libraries required by the 'tts-outetts.py' script.

```console
$ python3 -m venv venv
$ source venv/bin/activate
(venv) pip install requests numpy
```

--------------------------------

### Setup OpenCL Environment for Windows 11 Arm64

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/OPENCL.md

Commands to install OpenCL headers and the ICD loader on Windows 11 Arm64 using PowerShell.

```powershell
mkdir -p ~/dev/llm

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-Headers && cd OpenCL-Headers
mkdir build && cd build
cmake .. -G Ninja `
  -DBUILD_TESTING=OFF `
  -DOPENCL_HEADERS_BUILD_TESTING=OFF `
  -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF `
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader && cd OpenCL-ICD-Loader
mkdir build && cd build
cmake .. -G Ninja `
  -DCMAKE_BUILD_TYPE=Release `
  -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" `
  -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install
```

--------------------------------

### Quick Start - Interactive Mode

Source: https://github.com/tiiny-ai/powerinfer/blob/main/examples/main/README.md

Demonstrates how to run the 'main' program in interactive mode for chat-like interactions.

```APIDOC
## Interactive Mode Example

### Description
This example demonstrates how to run the 'main' program in interactive mode, allowing for a chat-like conversation with the LLaMA model.

### Method
Command Line Execution

### Endpoint
N/A (Local execution)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
#### Unix-based systems (Linux, macOS, etc.):
```bash
./main -m models/7B/ggml-model.bin -n -1 --color -r "User:" --in-prefix " " -i -p \
'User: Hi\nAI: Hello. I am an AI chatbot. Would you like to talk?\nUser: Sure!\nAI: What would you like to talk about?\nUser:'
```

#### Windows:
```powershell
main.exe -m models\7B\ggml-model.bin -n -1 --color -r "User:" --in-prefix " " -i -e -p "User: Hi\nAI: Hello. I am an AI chatbot. Would you like to talk?\nUser: Sure!\nAI: What would you like to talk about?\nUser:"
```

### Response
#### Success Response (200)
Interactive console for chat.

#### Response Example
(Interactive console session)
```

--------------------------------

### Initialize CLI11 Application

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/book/chapters/an-advanced-example.md

Sets up the base application and defines a required subcommand.

```cpp
CLI::App app{"geet"};
app.require_subcommand(1);
```

--------------------------------

### Get Chat Template Script

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/function-calling.md

Utilize the get_chat_template.py script to retrieve official chat templates from HuggingFace repositories. Refer to models/templates/README.md for example invocations.

```python
scripts/get_chat_template.py
```

--------------------------------

### C++ gMock Test Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/googletest/docs/gmock_cook_book.md

This C++ code demonstrates a basic gMock test setup with mock methods and expectations. It is used to illustrate the verbose output when run with specific flags.

```cpp
#include <gmock/gmock.h>

using ::testing::_;
using ::testing::HasSubstr;
using ::testing::Return;

class MockFoo {
 public:
  MOCK_METHOD(void, F, (const string& x, const string& y));
};

TEST(Foo, Bar) {
  MockFoo mock;
  EXPECT_CALL(mock, F(_, _)).WillRepeatedly(Return());
  EXPECT_CALL(mock, F("a", "b"));
  EXPECT_CALL(mock, F("c", HasSubstr("d")));

  mock.F("a", "good");
  mock.F("a", "b");
}
```

--------------------------------

### Initialize Project Directory

Source: https://github.com/tiiny-ai/powerinfer/blob/main/examples/server/README.md

Create and enter a new directory for the Node.js client project.

```bash
mkdir llama-client
cd llama-client
```

--------------------------------

### CLI11 CMake Build Options

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/book/chapters/installation.md

This table lists available CMake options for building CLI11. Options control features like single-file compilation, precompiled libraries, documentation, examples, installation, and compiler choices.

```cmake
CLI11_SINGLE_FILE=ON
```

```cmake
CLI11_PRECOMPILED=OFF
```

```cmake
CLI11_SINGLE_FILE_TESTS=OFF
```

```cmake
CLI11_BUILD_DOCS=ON
```

```cmake
CLI11_BUILD_EXAMPLES=ON
```

```cmake
CLI11_BUILD_EXAMPLES_JSON=ON
```

```cmake
CLI11_INSTALL=ON
```

```cmake
CLI11_FORCE_LIBCXX=OFF
```

```cmake
CLI11_CUDA_TESTS=OFF
```

```cmake
CLI11_BUILD_TESTS=ON
```

--------------------------------

### Start Server with Legacy UI

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/README.md

Command to launch the server using the legacy completion-based web interface.

```sh
./llama-server -m my_model.gguf -c 8192 --path ./tools/server/public_legacy
```

--------------------------------

### Verify GPU Driver Installation

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/SYCL.md

Install and run clinfo to verify the OpenCL driver installation.

```sh
sudo apt install clinfo
sudo clinfo -l
```

--------------------------------

### Example pkg-config file content

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/googletest/docs/pkgconfig.md

A sample structure of a generated .pc file for GoogleTest.

```text
libdir=/usr/lib64
includedir=/usr/include

Name: gtest
Description: GoogleTest (without main() function)
Version: 1.11.0
URL: https://github.com/google/googletest
Libs: -L${libdir} -lgtest -lpthread
Cflags: -I${includedir} -DGTEST_HAS_PTHREAD=1 -lpthread
```

--------------------------------

### Install CUDA Meta-Package

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/CUDA-FEDORA.md

Installs the CUDA toolkit and associated packages after the NVIDIA driver libraries have been successfully installed.

```bash
sudo dnf install cuda
```

--------------------------------

### Install Google Test Components

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/googletest/googletest/CMakeLists.txt

Installs the gtest and gtest_main targets. This rule is part of the project's installation process.

```cmake
install_project(gtest gtest_main)
```

--------------------------------

### Example Command Line Input

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/README.md

Illustrates a complex command line input string with options and nested subcommands to demonstrate callback execution.

```bash
program --opt1 opt1_val  sub1 --sub1opt --sub1optb val sub2 --sub2opt sub1 --sub1opt2 sub2 --sub2opt2 val
```

--------------------------------

### Install Metal Files

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/ggml/src/ggml-metal/CMakeLists.txt

Installs the source Metal shader file and the compiled metallib file to the designated installation directory.

```cmake
install(
    FILES src/ggml-metal/ggml-metal.metal
    PERMISSIONS
        OWNER_READ
        OWNER_WRITE
        GROUP_READ
        WORLD_READ
    DESTINATION ${CMAKE_INSTALL_BINDIR})

    install(
        FILES ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/default.metallib
        DESTINATION ${CMAKE_INSTALL_BINDIR}
    )
```

--------------------------------

### Load Model and Create Context with C API

Source: https://context7.com/tiiny-ai/powerinfer/llms.txt

Demonstrates initializing the backend, configuring model and context parameters, loading a model from a file, creating a context, and retrieving basic model information. Ensure the model file exists at the specified path. Cleanup is essential after use.

```c
#include "llama.h"

int main() {
    // Initialize backend (call once at program start)
    llama_backend_init(false);  // false = no NUMA optimizations

    // Configure model parameters
    struct llama_model_params model_params = llama_model_default_params();
    model_params.n_gpu_layers = 35;        // Layers to offload to GPU
    model_params.vram_budget_gb = 8.0f;    // Limit VRAM usage
    model_params.use_mmap = true;          // Memory-map for faster loading
    model_params.use_mlock = false;        // Don't lock in RAM

    // Load the model
    struct llama_model *model = llama_load_model_from_file(
        "./model.powerinfer.gguf",
        model_params
    );
    if (model == NULL) {
        fprintf(stderr, "Failed to load model\n");
        return 1;
    }

    // Configure context parameters
    struct llama_context_params ctx_params = llama_context_default_params();
    ctx_params.seed = 1234;
    ctx_params.n_ctx = 2048;           // Context window size
    ctx_params.n_batch = 512;          // Batch size for prompt processing
    ctx_params.n_threads = 8;          // Threads for generation
    ctx_params.n_threads_batch = 8;    // Threads for batch processing
    ctx_params.f16_kv = true;          // Use FP16 for KV cache

    // Create context
    struct llama_context *ctx = llama_new_context_with_model(model, ctx_params);
    if (ctx == NULL) {
        fprintf(stderr, "Failed to create context\n");
        llama_free_model(model);
        return 1;
    }

    // Get model info
    printf("Vocab size: %d\n", llama_n_vocab(model));
    printf("Context size: %d\n", llama_n_ctx(ctx));
    printf("Embedding size: %d\n", llama_n_embd(model));

    // Cleanup
    llama_free(ctx);
    llama_free_model(model);
    llama_backend_free();

    return 0;
}

```

--------------------------------

### Install Android NDK

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/OPENCL.md

Installs the Android NDK version 26.3.11579264 using the command line tools. Ensure you have wget and unzip installed.

```sh
cd ~
wget https://dl.google.com/android/repository/commandlinetools-linux-8512546_latest.zip && \
unzip commandlinetools-linux-8512546_latest.zip && \
mkdir -p ~/android-sdk/cmdline-tools && \
mv cmdline-tools latest && \
mv latest ~/android-sdk/cmdline-tools/ && \
rm -rf commandlinetools-linux-8512546_latest.zip

yes | ~/android-sdk/cmdline-tools/latest/bin/sdkmanager "ndk;26.3.11579264"
```

--------------------------------

### Quick Start - Basic Inference

Source: https://github.com/tiiny-ai/powerinfer/blob/main/examples/main/README.md

Demonstrates how to run a basic inference with a LLaMA model using the 'main' program.

```APIDOC
## Basic Inference Example

### Description
This example shows how to perform a simple text generation task using a specified LLaMA model and prompt.

### Method
Command Line Execution

### Endpoint
N/A (Local execution)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
#### Unix-based systems (Linux, macOS, etc.):
```bash
./main -m models/7B/ggml-model.bin --prompt "Once upon a time"
```

#### Windows:
```powershell
main.exe -m models\7B\ggml-model.bin --prompt "Once upon a time"
```

### Response
#### Success Response (200)
Output will be the generated text based on the prompt.

#### Response Example
(Output depends on the model and prompt)
```

--------------------------------

### Build and Run Swift Batched Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/examples/batched.swift/README.md

Commands to compile the project and execute the batched inference binary with model path, prompt, and parallel parameters.

```bash
$ make
```

```bash
$ ./llama-batched-swift MODEL_PATH [PROMPT] [PARALLEL]
```

--------------------------------

### Add Subdirectories for Examples (Non-Emscripten)

Source: https://github.com/tiiny-ai/powerinfer/blob/main/examples/CMakeLists.txt

Adds various example subdirectories to the build when not targeting Emscripten. This includes examples for different inference techniques, benchmarks, and model conversion.

```cmake
add_subdirectory(baby-llama)
```

```cmake
add_subdirectory(batched)
```

```cmake
add_subdirectory(batched-bench)
```

```cmake
add_subdirectory(beam-search)
```

```cmake
add_subdirectory(benchmark)
```

```cmake
add_subdirectory(convert-llama2c-to-ggml)
```

```cmake
add_subdirectory(embedding)
```

```cmake
add_subdirectory(finetune)
```

```cmake
add_subdirectory(infill)
```

```cmake
add_subdirectory(llama-bench)
```

```cmake
add_subdirectory(llava)
```

```cmake
add_subdirectory(main)
```

```cmake
add_subdirectory(parallel)
```

```cmake
add_subdirectory(perplexity)
```

```cmake
add_subdirectory(quantize)
```

```cmake
add_subdirectory(quantize-stats)
```

```cmake
add_subdirectory(save-load-state)
```

```cmake
add_subdirectory(simple)
```

```cmake
add_subdirectory(speculative)
```

```cmake
add_subdirectory(train-text-from-scratch)
```

--------------------------------

### Sample clinfo Output

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/SYCL.md

Example output showing detected Intel GPU devices.

```text
Platform #0: Intel(R) OpenCL Graphics
 `-- Device #0: Intel(R) Arc(TM) A770 Graphics

Platform #0: Intel(R) OpenCL HD Graphics
 `-- Device #0: Intel(R) Iris(R) Xe Graphics [0x9a49]
```

--------------------------------

### Install BLIS

Source: https://github.com/tiiny-ai/powerinfer/blob/main/docs/BLIS.md

Install the compiled BLIS library to the system path.

```bash
sudo make install
```

--------------------------------

### Setting Up Callbacks for CLI Parsing in C++

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/README.md

Demonstrates the order of callback execution for a main application with subcommands and options. Callbacks include preparse, parse_complete, and final.

```cpp
app.parse_complete_callback(ac1);
app.final_callback(ac2);
auto sub1=app.add_subcommand("sub1")->parse_complete_callback(c1)->preparse_callback(pc1);
auto sub2=app.add_subcommand("sub2")->final_callback(c2)->preparse_callback(pc2);
app.preparse_callback( pa);
```

--------------------------------

### Run Multimodal Models via CLI and Server

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/multimodal.md

Examples of launching multimodal tools with various configurations for model loading and GPU offloading.

```sh
# simple usage with CLI
llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF

# simple usage with server
llama-server -hf ggml-org/gemma-3-4b-it-GGUF

# using local file
llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf

# no GPU offload
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --no-mmproj-offload
```

--------------------------------

### Print Hello World to Stdout with fmt

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/README.md

Use fmt::print to output strings directly to the standard output. Requires including the <fmt/base.h> header.

```c++
#include <fmt/base.h>

int main() {
  fmt::print("Hello, world!\n");
}
```

--------------------------------

### Install Python Requirements

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/multimodal/llava.md

Install the necessary Python dependencies for conversion tools.

```sh
pip install -r tools/mtmd/requirements.txt
```

--------------------------------

### Simple Format String Examples

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/doc/syntax.md

Illustrates basic usage of format strings with explicit argument indexing and implicit sequential indexing.

```c++
"First, thou shalt count to {0}"
```

```c++
"Bring me a {}"
```

```c++
"From {} to {}"
```

--------------------------------

### Install k6 with SSE extension

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/bench/README.md

Builds a custom k6 binary with SSE support using xk6. Requires Go 1.21 or higher.

```shell
go install go.k6.io/xk6/cmd/xk6@latest
$GOPATH/bin/xk6 build master \
--with github.com/phymbert/xk6-sse
```

--------------------------------

### Install GGUF package

Source: https://github.com/tiiny-ai/powerinfer/blob/main/gguf-py/README.md

Standard installation command for the GGUF Python package.

```sh
pip install gguf
```

--------------------------------

### Run Passkey Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/examples/passkey/README.md

Compile and run the llama-passkey example. Requires a GGML model file. The --junk argument specifies the amount of random data to generate.

```bash
make -j && ./llama-passkey -m ./models/llama-7b-v2/ggml-model-f16.gguf --junk 250
```

--------------------------------

### Installed CMake Target

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/doc/get-started.md

Find and use an installed version of {fmt} in your CMakeLists.txt file.

```cmake
find_package(fmt)
target_link_libraries(<your-target> fmt::fmt)
```

--------------------------------

### Install GGUF with GUI support

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/gguf-py/README.md

Installs the package with the optional GUI dependencies enabled.

```sh
pip install gguf[gui]
```

--------------------------------

### Build CLI11 with Meson and Ninja

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/tests/mesonTest/README.md

Use these commands to build the project. Ensure Meson and Ninja are installed.

```bash
meson build
```

```bash
ninja -C build
```

--------------------------------

### Android Inference Examples

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/multimodal/MobileVLM.md

Example commands and output for running inference on an Android device.

```sh
/data/local/tmp/llama-mtmd-cli \
    -m /data/local/tmp/ggml-model-q4_k.gguf \
    --mmproj /data/local/tmp/mmproj-model-f16.gguf \
    -t 4 \
    --image /data/local/tmp/demo.jpg \
    -p "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWho is the author of this book? \nAnswer the question using a single word or phrase. ASSISTANT:"
```

```sh
encode_image_with_clip: image encoded in 21148.71 ms by CLIP (  146.87 ms per image patch)
 Susan Wise Bauer
llama_print_timings:        load time =   23574.72 ms
llama_print_timings:      sample time =       1.24 ms /     6 runs   (    0.21 ms per token,  4850.44 tokens per second)
llama_print_timings: prompt eval time =   12460.15 ms /   246 tokens (   50.65 ms per token,    19.74 tokens per second)
llama_print_timings:        eval time =     424.86 ms /     6 runs   (   70.81 ms per token,    14.12 tokens per second)
llama_print_timings:       total time =   34731.93 ms
```

```sh
/data/local/tmp/llama-mtmd-cli \
    -m /data/local/tmp/ggml-model-q4_k.gguf \
    --mmproj /data/local/tmp/mmproj-model-f16.gguf \
    -t 4 \
    --image /data/local/tmp/cat.jpeg \
    -p "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat is in the image? ASSISTANT:"
```

```sh
encode_image_with_clip: image encoded in 21149.51 ms by CLIP (  146.87 ms per image patch)
 The image depicts a cat sitting in the grass near some tall green plants.
llama_print_timings:        load time =   23257.32 ms
llama_print_timings:      sample time =       5.25 ms /    18 runs   (    0.29 ms per token,  3430.53 tokens per second)
llama_print_timings: prompt eval time =   11900.73 ms /   232 tokens (   51.30 ms per token,    19.49 tokens per second)
llama_print_timings:        eval time =    1279.03 ms /    18 runs   (   71.06 ms per token,    14.07 tokens per second)
llama_print_timings:       total time =   34570.79 ms
```

--------------------------------

### llama-embedding Advanced Examples

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/examples/embedding/README.md

Examples demonstrating the use of advanced parameters for generating embeddings.

```APIDOC
## Advanced Examples

These examples showcase the usage of various parameters for generating embeddings with custom separators and normalization.

### Unix-based systems (Linux, macOS, etc.):
```bash
./llama-embedding -p 'Castle<#sep#>Stronghold<#sep#>Dog<#sep#>Cat' --pooling mean --embd-separator '<#sep#>' --embd-normalize 2  --embd-output-format '' -m './path/to/model.gguf' --n-gpu-layers 99 --log-disable 2>/dev/null
```

### Windows:
```powershell
llama-embedding.exe -p 'Castle<#sep#>Stronghold<#sep#>Dog<#sep#>Cat' --pooling mean --embd-separator '<#sep#>' --embd-normalize 2  --embd-output-format '' -m './path/to/model.gguf' --n-gpu-layers 99 --log-disable 2>/dev/null
```
```

--------------------------------

### Experimental fmt::writer API Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/ChangeLog.md

Demonstrates the usage of the experimental `fmt::writer` API for writing to different destinations like stdout, files, and strings. Ensure necessary headers are included.

```c++
#include <fmt/os.h>

void write_text(fmt::writer w) {
  w.print("The answer is {}.", 42);
}

int main() {
  // Write to FILE.
  write_text(stdout);

  // Write to fmt::ostream.
  auto f = fmt::output_file("myfile");
  write_text(f);

  // Write to std::string.
  auto sb = fmt::string_buffer();
  write_text(sb);
  std::string s = sb.str();
}
```

--------------------------------

### Initialize oneAPI Environment

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/backend/SYCL.md

Run this command in the oneAPI command line window to set up the environment before working with llama.cpp.

```batch
"C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64
```

--------------------------------

### Install {fmt} via Homebrew

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/fmt/doc/get-started.md

Install {fmt} on macOS using the Homebrew package manager.

```bash
brew install fmt
```

--------------------------------

### Launch llama-server with native template support

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/function-calling.md

Start the server using models with native Jinja template support.

```shell
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
llama-server --jinja -fa -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K_L
llama-server --jinja -fa -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M
```

--------------------------------

### Install llama.cpp via MacPorts

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/docs/install.md

Use this command on Mac to install the package via MacPorts.

```sh
sudo port install llama.cpp
```

--------------------------------

### Install GGUF in editable mode

Source: https://github.com/tiiny-ai/powerinfer/blob/main/gguf-py/README.md

Commands to install the package in editable mode for development purposes.

```sh
cd /path/to/llama.cpp/gguf-py

pip install --editable .
```

--------------------------------

### INI Configuration Example

Source: https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/libaz/external/cli11/book/chapters/config.md

Demonstrates INI file format for CLI11 configuration, including comments, default sections, and mapping sections to subcommands.

```ini
; Comments are supported, using a ;
; The default section is [default], case-insensitive

value = 1
str = "A string"
vector = 1 2 3

; Section map to subcommands
[subcommand]
in_subcommand = Wow
sub.subcommand = true

```