### Install Git on Raspberry Pi

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md

Install the Git version control system on all Raspberry Pi devices to clone the repository.

```bash
sudo apt install git
```

--------------------------------

### Install Git and Mingw on Windows

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_LINUX_MACOS_WIN.md

Installs Git and the MinGW development environment on Windows using Chocolatey.

```powershell
choco install git mingw
```

--------------------------------

### Install Git on MacOS

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_LINUX_MACOS_WIN.md

Installs Git on macOS using the Homebrew package manager.

```shell
brew install git
```

--------------------------------

### Install Prerequisites and Build Distributed Llama (CPU)

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Installs necessary build tools for Linux, macOS, or Windows, then clones the repository and builds the CPU versions of `dllama` and `dllama-api` using `make`.

```sh
# Install prerequisites (Linux)
sudo apt install git build-essential

# Install prerequisites (macOS)
brew install git

# Install prerequisites (Windows via Chocolatey)
choco install git mingw

# Clone and build CPU binaries
git clone https://github.com/b4rtaz/distributed-llama.git
cd distributed-llama
make dllama
make dllama-api
```

--------------------------------

### Start Multiple Local Test Workers using n-workers.sh

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Starts multiple worker nodes for local testing using the `n-workers.sh` script. This example starts 3 workers on ports 9999, 9998, and 9997, each using 2 threads.

```sh
# Start multiple workers for local testing (uses screen)
# W=3 workers on ports 9999, 9998, 9997 with T=2 threads each
W=3 T=2 bash examples/n-workers.sh start
```

--------------------------------

### Setup Root Node with Model Download

Source: https://github.com/b4rtaz/distributed-llama/blob/main/README.md

Use this command to set up the root node, which automatically downloads the specified model and tokenizer. Python 3 and a C++ compiler are required.

```bash
python launch.py llama3_1_8b_instruct_q40
```

```bash
python launch.py llama3_1_405b_instruct_q40
```

```bash
python launch.py llama3_2_1b_instruct_q40
```

```bash
python launch.py llama3_2_3b_instruct_q40
```

```bash
python launch.py llama3_3_70b_instruct_q40
```

```bash
python launch.py deepseek_r1_distill_llama_8b_q40
```

```bash
python launch.py qwen3_0.6b_q40
```

```bash
python launch.py qwen3_1.7b_q40
```

```bash
python launch.py qwen3_8b_q40
```

```bash
python launch.py qwen3_14b_q40
```

```bash
python launch.py qwen3_30b_a3b_q40
```

--------------------------------

### Install Git and Build Tools on Linux

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_LINUX_MACOS_WIN.md

Installs Git and essential build tools for compiling C++ applications on Debian-based Linux systems.

```shell
sudo apt install git build-essential
```

--------------------------------

### Start API Server (Single Device)

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Launch the dllama API server on a specified port for single-device operation. This provides an OpenAI-compatible endpoint.

```bash
./dllama-api \
  --port 9999 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096
```

--------------------------------

### API Server

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Start an OpenAI-compatible API server for model inference.

```APIDOC
## ./dllama-api

### Description
Exposes an OpenAI-compatible HTTP API for model inference, supporting streaming and tool calling.

### Method
Command-line execution

### Endpoint
N/A (Command-line tool, serves HTTP API)

### Parameters
- `--port` (integer) - Required - The port to run the API server on.
- `--host` (string) - Optional - The host address to bind to (default: localhost).
- `--model` (string) - Required - Path to the model file.
- `--tokenizer` (string) - Required - Path to the tokenizer file.
- `--buffer-float-type` (string) - Required - The float type for buffers (e.g., q80).
- `--nthreads` (integer) - Required - Number of threads to use.
- `--max-seq-len` (integer) - Optional - Maximum sequence length.
- `--gpu-index` (integer) - Optional - The index of the GPU to use for GPU-accelerated builds.
- `--workers` (list of strings) - Optional - List of worker addresses for distributed API server.

### Request Example (Single-device)
```sh
./dllama-api \
  --port 9999 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096
```

### Request Example (Distributed)
```sh
./dllama-api \
  --host 0.0.0.0 \
  --port 9999 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
```
```

--------------------------------

### Start Distributed Llama API Server

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_LINUX_MACOS_WIN.md

Starts the API server on the root device, configured to use specified worker nodes for distributed processing.

```shell
./dllama-api \
  --port 9999 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
```

--------------------------------

### Start Distributed Llama API Server

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md

Start the Distributed Llama API server on the root Raspberry Pi device. Configure host, port, model, tokenizer, and worker details.

```bash
sudo nice -n -20 ./dllama-api \
  --host 0.0.0.0 \
  --port 9999 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
```

--------------------------------

### Start Distributed Llama API Server (Root Node)

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Launch the API server on the root node after worker nodes are running. Specify model, tokenizer, and worker addresses. Ensure correct paths for model and tokenizer files.

```bash
sudo nice -n -20 ./dllama-api \
  --host 0.0.0.0 --port 9999 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 --nthreads 4 --max-seq-len 4096 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
```

--------------------------------

### Start Distributed Llama Workers

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md

Start the Distributed Llama worker process on each worker Raspberry Pi device. Specify the port and number of threads.

```bash
sudo nice -n -20 ./dllama worker --port 9999 --nthreads 4
```

--------------------------------

### Start Distributed Llama Worker

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_LINUX_MACOS_WIN.md

Starts a worker process on a designated device, listening on a specified port and using a set number of threads.

```shell
./dllama worker --port 9999 --nthreads 4
```

--------------------------------

### POST /v1/chat/completions (Tool Calling)

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Supports tool calling by providing `tools` in the request. The API injects a system prompt to guide the model in emitting JSON tool calls, which are then parsed and populated in the `tool_calls` field of the response when `finish_reason` is `tool_calls`.

```APIDOC
## POST /v1/chat/completions (Tool Calling)

### Description
This endpoint facilitates tool calling. When `tools` are provided in the request, the model can generate JSON output to invoke specified functions. The API parses this output and includes it in the `tool_calls` field of the response.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Request Body
- **messages** (array) - Required - An array of message objects.
- **tools** (array) - Optional - A list of tools the model may call.
  - **type** (string) - Required - The type of tool (e.g., `function`).
  - **function** (object) - Required - Defines the function to be called.
    - **name** (string) - Required - The name of the function.
    - **description** (string) - Optional - A description of the function.
    - **parameters** (object) - Optional - The parameters the function accepts.
      - **type** (string) - Required - The type of the parameters object (e.g., `object`).
      - **properties** (object) - Optional - Definitions of the parameters.
      - **required** (array of strings) - Optional - List of required parameter names.
- **tool_choice** (string or object) - Optional - Controls how the model chooses a tool. `auto` is common.
- **max_tokens** (integer) - Optional - Maximum tokens to generate.

### Request Example
```sh
curl -X POST http://localhost:9999/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You can use only 1 tool at a time."},
      {"role": "user",   "content": "Tell me about the most popular car from Toyota and its sales this year."}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_most_popular_car_by_company",
        "description": "Return the most popular car model for a given company name.",
        "parameters": {
          "type": "object",
          "properties": {
            "companyName": {"type": "string", "description": "Car company name, e.g., Toyota"}
          },
          "required": ["companyName"]
        }
      }
    }],
    "tool_choice": "auto",
    "max_tokens": 256
  }'
```

### Response
#### Success Response (when model decides to call a tool)
- **id** (string) - Unique identifier for the completion.
- **object** (string) - Type of object, e.g., `chat.completion`.
- **choices** (array) - List of completion choices.
  - **index** (integer) - Index of the choice.
  - **message** (object) - The message object.
    - **role** (string) - Role of the message sender (`assistant`).
    - **content** (string) - Usually empty when `tool_calls` are present.
    - **tool_calls** (array) - List of tool calls made by the model.
      - **id** (string) - Unique identifier for the tool call.
      - **type** (string) - Type of tool call (e.g., `function`).
      - **function** (object) - Details of the function call.
        - **name** (string) - The name of the function to call.
        - **arguments** (string) - JSON string representing the arguments for the function call.
  - **finish_reason** (string) - The reason the generation stopped (e.g., `tool_calls`).

#### Response Example (when model decides to call a tool):
```json
{
  "id": "cmpl-j0",
  "object": "chat.completion",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "",
      "tool_calls": [{
        "id": "call_1",
        "type": "function",
        "function": {
          "name": "get_most_popular_car_by_company",
          "arguments": "{\"companyName\": \"Toyota\"}"
        }
      }]
    },
    "finish_reason": "tool_calls"
  }],
  ...
}
```
```

--------------------------------

### Build Distributed Llama with Vulkan GPU Support

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Builds `dllama` and `dllama-api` with Vulkan GPU acceleration enabled. Requires the Vulkan SDK to be installed beforehand. The `DLLAMA_VULKAN=1` flag is used during the `make` process.

```sh
# Build with Vulkan GPU support
DLLAMA_VULKAN=1 make dllama
DLLAMA_VULKAN=1 make dllama-api
```

--------------------------------

### Start a Distributed Llama Worker Node

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Starts a worker node for the Distributed Llama cluster. Workers listen on a specified TCP port and utilize a given number of CPU threads. GPU acceleration requires a Vulkan build and uses a single thread.

```sh
# Start a worker on port 9999 using 4 CPU threads
./dllama worker --port 9999 --nthreads 4

# Start a worker bound to a specific interface
./dllama worker --host 0.0.0.0 --port 9999 --nthreads 4

# Worker with GPU acceleration (Vulkan build required, single thread mandatory)
./dllama worker --port 9999 --nthreads 1 --gpu-index 0

# Raspberry Pi (elevate priority for better performance)
sudo nice -n -20 ./dllama worker --port 9999 --nthreads 4
```

--------------------------------

### Run Distributed Llama on GPU

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_GPU.md

Execute Distributed Llama binaries on a GPU device. Ensure Vulkan is installed and the project is built with GPU support. Use `--nthreads 1` as the Vulkan backend requires a single thread.

```bash
./dllama inference ... --nthreads 1 --gpu-index 0 
```

```bash
./dllama chat      ... --nthreads 1 --gpu-index 0 
```

```bash
./dllama worker    ... --nthreads 1 --gpu-index 0 
```

```bash
./dllama-api       ... --nthreads 1 --gpu-index 0 
```

--------------------------------

### REST API - GET /v1/models

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Retrieves a list of currently loaded models in OpenAI format.

```APIDOC
## GET /v1/models

### Description
Returns the list of currently loaded models in OpenAI format.

### Method
GET

### Endpoint
`/v1/models`

### Parameters
None

### Request Example
```sh
curl http://localhost:9999/v1/models
```

### Response
#### Success Response (200)
- **object** (string) - The type of object returned, "list".
- **data** (array) - An array of model objects.
  - **id** (string) - The unique identifier of the model.
  - **object** (string) - The type of object, "model".
  - **created** (integer) - Timestamp of model creation.
  - **owned_by** (string) - The owner of the model.

#### Response Example
```json
{
  "object": "list",
  "data": [
    {
      "id": "dllama_model_llama3_2_3b_instruct_q40.m",
      "object": "model",
      "created": 0,
      "owned_by": "user"
    }
  ]
}
```
```

--------------------------------

### Get Loaded Models via REST API

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Query the /v1/models endpoint of the dllama API server to retrieve a list of currently loaded models in OpenAI format.

```bash
curl http://localhost:9999/v1/models
```

--------------------------------

### Stop Local Test Workers

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Stops all local test workers that were started using the `n-workers.sh` script. This command is used to clean up testing environments.

```sh
# Stop all local test workers
W=3 bash examples/n-workers.sh stop
```

--------------------------------

### Download and List Available Models with launch.py

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Uses the `launch.py` script to list all supported models available for download from HuggingFace. Models are downloaded to the local `models/` directory.

```python
# List all available models
python3 launch.py
```

--------------------------------

### Download Model using launch.py

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_LINUX_MACOS_WIN.md

Downloads a specified language model to the root device. Use the first command to list available models.

```python
python3 launch.py # Prints a list of available models
```

```python
python3 launch.py llama3_2_3b_instruct_q40 # Downloads the model to the root device
```

--------------------------------

### Download a Specific Model with launch.py

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Downloads a specified quantized model and its tokenizer from HuggingFace. The script can optionally skip running the model afterward or skip creating a run script and confirmation prompts.

```python
# Download Llama 3.2 3B Instruct Q40
python3 launch.py llama3_2_3b_instruct_q40

# Download without running afterward
python3 launch.py llama3_2_3b_instruct_q40 -skip-run

# Download without creating a run script and skip confirmation prompts
python3 launch.py llama3_3_70b_instruct_q40 -skip-script -y
```

--------------------------------

### API Server with GPU Support (Vulkan)

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Enable GPU acceleration for the API server using a Vulkan build. Specify the GPU index and number of threads.

```bash
./dllama-api \
  --port 9999 \
  --nthreads 1 --gpu-index 0 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80
```

--------------------------------

### JavaScript API Client for Agentic Tool-Calling

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Demonstrates a full agentic tool-calling loop using the Node.js fetch API. Configure via environment variables for host and port.

```javascript
// Run with: node examples/chat-api-client.js
// Configure via env: HOST=10.0.0.1 PORT=9999 DEBUG=1 node examples/chat-api-client.js

const HOST = process.env.HOST ?? '127.0.0.1';
const PORT = Number(process.env.PORT ?? 9990);

async function complete(messages, maxTokens, extra = {}) {
    const response = await fetch(`http://${HOST}:${PORT}/v1/chat/completions`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
            messages,
            temperature: 0.7,
            stop: ['<|eot_id|>'],
            max_tokens: maxTokens,
            ...extra
        }),
    });
    return response.json();
}

// Simple Q&A
async function ask(system, user, maxTokens) {
    const response = await complete([
        { role: 'system', content: system },
        { role: 'user',   content: user }
    ], maxTokens);
    console.log(response.choices[0].message.content);
}

// Multi-turn agentic loop with tool calls
async function askWithTools(companyName, maxTokens) {
    const tools = [{
        type: 'function',
        function: {
            name: 'get_most_popular_car_by_company',
            description: 'Return the most popular car model for a given company.',
            parameters: {
                type: 'object',
                properties: { companyName: { type: 'string' } },
                required: ['companyName']
            }
        }
    }];

    const messages = [
        { role: 'system', content: 'You can use only 1 tool at a time.' },
        { role: 'user',   content: `Most popular car from ${companyName}?` }
    ];

    for (;;) {
        const response = await complete(messages, maxTokens, { tools, tool_choice: 'auto' });
        const choice = response.choices[0];
        if (choice.finish_reason !== 'tool_calls') break;

        messages.push(choice.message);
        for (const call of choice.message.tool_calls) {
            messages.push({
                role: 'tool',
                tool_call_id: call.id,
                content: JSON.stringify({ carName: 'Corolla' })  // simulated result
            });
        }
    }
    console.log(response.choices[0].message.content);
}

await ask('You are a math teacher.', 'What is 1 + 2?', 256);
await askWithTools('Toyota', 5000);

```

--------------------------------

### Convert Hugging Face Tokenizer

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_CONVERT_HF_MODEL.md

Run this script to convert the tokenizer files from a Hugging Face model. Provide the path to the model and the desired tokenizer name.

```sh
python convert-tokenizer-hf.py path/to/hf/model mistral-7b-0.3
```

--------------------------------

### GPU-Accelerated Inference (Vulkan)

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Perform inference using GPU acceleration with a Vulkan build. Specify the GPU index and use a single thread for optimal performance.

```bash
./dllama inference \
  --prompt "Hello world" \
  --steps 64 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 1 --gpu-index 0
```

--------------------------------

### Build Distributed Llama with GPU Support

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_GPU.md

Compile the project with Vulkan support enabled for GPU usage. This is required before running any GPU-accelerated commands.

```bash
DLLAMA_VULKAN=1 make dllama
DLLAMA_VULKAN=1 make dllama-api
```

--------------------------------

### Chat on Raspberry Pi with Elevated Priority

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Run the chat command on a Raspberry Pi using `nice` to elevate process priority. This can improve performance on resource-constrained devices.

```bash
sudo nice -n -20 ./dllama chat \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4
```

--------------------------------

### Download Llama Model

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md

Download the desired Llama model to the root Raspberry Pi device using the launch.py script. Worker devices do not require the model.

```python
python3 launch.py # Prints a list of available models

python3 launch.py llama3_2_3b_instruct_q40 # Downloads the model to the root device
```

--------------------------------

### Run Single-Device Inference with dllama

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Performs a single forward-pass generation from a text prompt using the `dllama inference` command. It prints per-token benchmark statistics and requires specifying the model and tokenizer files, along with buffer type and thread count.

```sh
# Single-device inference (no workers)
./dllama inference \
  --prompt "Hello world" \
  --steps 64 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4
```

--------------------------------

### Tool Calling via Chat Completion

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Utilize tool calling by providing a `tools` array in the request. The API will inject a system prompt, and if the model decides to use a tool, `tool_calls` will be populated in the response.

```sh
curl -X POST http://localhost:9999/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You can use only 1 tool at a time."},
      {"role": "user",   "content": "Tell me about the most popular car from Toyota and its sales this year."}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_most_popular_car_by_company",
        "description": "Return the most popular car model for a given company name.",
        "parameters": {
          "type": "object",
          "properties": {
            "companyName": {"type": "string", "description": "Car company name, e.g., Toyota"}
          },
          "required": ["companyName"]
        }
      }
    }],
    "tool_choice": "auto",
    "max_tokens": 256
  }'
```

--------------------------------

### Single-Device Chat

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Launch an interactive chat session on a single device. Requires model and tokenizer paths.

```bash
./dllama chat \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096
```

--------------------------------

### POST /v1/chat/completions (Non-streaming)

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Provides standard chat completions similar to OpenAI's API. It supports parameters like temperature, max_tokens, seed, stop, and tool_choice for fine-tuning the response.

```APIDOC
## POST /v1/chat/completions (Non-streaming)

### Description
This endpoint provides standard chat completions, supporting various parameters to control the generation process. It is compatible with the OpenAI API.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Request Body
- **messages** (array) - Required - An array of message objects, each with a `role` (system, user, or assistant) and `content`.
- **temperature** (number) - Optional - Controls randomness. Lower values make output more focused and deterministic.
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate in the completion.
- **seed** (integer) - Optional - A seed for reproducible results.
- **stop** (array of strings) - Optional - Sequences where the API will stop generating further tokens.
- **tool_choice** (string or object) - Optional - Controls how tools are invoked.

### Request Example
```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."}, 
    {"role": "user",   "content": "What is the capital of France?"}
  ],
  "temperature": 0.7,
  "max_tokens": 128,
  "seed": 42,
  "stop": ["<|eot_id|>"]
}
```

### Response
#### Success Response (200)
- **id** (string) - Unique identifier for the completion.
- **object** (string) - Type of object, e.g., `chat.completion`.
- **created** (integer) - Timestamp of creation.
- **model** (string) - The model used for completion.
- **usage** (object) - Token usage statistics.
  - **prompt_tokens** (integer) - Tokens in the prompt.
  - **completion_tokens** (integer) - Tokens in the completion.
  - **total_tokens** (integer) - Total tokens used.
- **choices** (array) - List of completion choices.
  - **index** (integer) - Index of the choice.
  - **message** (object) - The generated message.
    - **role** (string) - Role of the message sender (e.g., `assistant`).
    - **content** (string) - The content of the message.
  - **finish_reason** (string) - The reason the generation stopped (e.g., `stop`).

#### Response Example
```json
{
  "id": "cmpl-j0",
  "object": "chat.completion",
  "created": 1718000000,
  "model": "Distributed Model",
  "usage": {
    "prompt_tokens": 24,
    "completion_tokens": 12,
    "total_tokens": 36
  },
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "The capital of France is Paris."},
    "finish_reason": "stop"
  }]
}
```
```

--------------------------------

### Distributed API Server with 3 Workers

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Configure the API server to run in a distributed mode across multiple workers. Specify the host and port for the server.

```bash
./dllama-api \
  --host 0.0.0.0 \
  --port 9999 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
```

--------------------------------

### Run Inference with Converted Model

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_CONVERT_HF_MODEL.md

After converting the model and tokenizer, use this command to run inference with Distributed Llama. Specify the prompt, steps, and the paths to your converted model and tokenizer files.

```sh
./dllama inference \
  --prompt "Hello world" \
  --steps 64 \
  --model dllama_model_mistral-7b-0.3_q40.m \
  --tokenizer dllama_tokenizer_mistral-7b-0.3.t \
  --buffer-float-type q80
```

--------------------------------

### Connect to Raspberry Pi Devices via SSH

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md

Connect to each Raspberry Pi device using SSH. Ensure you have the correct username and hostname or IP address.

```bash
ssh user@raspberrypi1.local
ssh user@raspberrypi2.local
ssh user@raspberrypi3.local
ssh user@raspberrypi4.local
```

--------------------------------

### Assign Static IPs on Linux/Raspberry Pi

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Assign static IP addresses to network interfaces for cluster nodes. Ensure 'eth0' is the correct network interface.

```bash
sudo ip addr add 10.0.0.1/24 dev eth0   # Root node
```

```bash
sudo ip addr add 10.0.0.2/24 dev eth0   # Worker 1
```

```bash
sudo ip addr add 10.0.0.3/24 dev eth0   # Worker 2
```

```bash
sudo ip addr add 10.0.0.4/24 dev eth0   # Worker 3
```

--------------------------------

### Streaming Chat Completion

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Enable streaming by setting `"stream": true` to receive Server-Sent Events (SSE) chunks. This is useful for real-time applications. Ensure `--no-buffer` is used with curl for immediate output.

```sh
curl -X POST http://localhost:9999/v1/chat/completions \
  -H "Content-Type: application/json" \
  --no-buffer \
  -d '{
    "messages": [
      {"role": "user", "content": "Count from 1 to 5."}
    ],
    "temperature": 0.5,
    "max_tokens": 64,
    "stream": true
  }'
```

--------------------------------

### Build Distributed Llama with Debug/Sanitizer Flags

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Builds the `dllama` executable with debug and sanitizer flags enabled for development and testing purposes. The `DEBUG=1` flag is used with `make`.

```sh
# Build with debug/sanitizer flags
DEBUG=1 make dllama
```

--------------------------------

### POST /v1/chat/completions (Streaming)

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Enables streaming chat completions by setting the `stream` parameter to `true`. Responses are delivered as Server-Sent Events (SSE) chunks, with the stream ending with `data: [DONE]`.

```APIDOC
## POST /v1/chat/completions (Streaming)

### Description
This endpoint provides chat completions in a streaming format using Server-Sent Events (SSE). Set `"stream": true` in the request body to enable this feature.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Request Body
- **messages** (array) - Required - An array of message objects.
- **temperature** (number) - Optional - Controls randomness.
- **max_tokens** (integer) - Optional - Maximum tokens to generate.
- **stream** (boolean) - Required - Set to `true` to enable streaming.

### Request Example
```sh
curl -X POST http://localhost:9999/v1/chat/completions \
  -H "Content-Type: application/json" \
  --no-buffer \
  -d '{
    "messages": [
      {"role": "user", "content": "Count from 1 to 5."}
    ],
    "temperature": 0.5,
    "max_tokens": 64,
    "stream": true
  }'
```

### Response
#### Success Response (SSE Stream)
Responses are delivered as SSE chunks. Each chunk starts with `data: ` followed by a JSON object representing a part of the completion. The stream concludes with `data: [DONE]`.

#### Response Example (SSE stream):
```
data: {"id":"cmpl-c0","object":"chat.completion","created":1718000001,"model":"Distributed Model","choices":[{"index":0,"delta":{"role":"assistant","content":"1"},"finish_reason":""}]}

data: {"id":"cmpl-c0","object":"chat.completion","created":1718000001,"model":"Distributed Model","choices":[{"index":0,"delta":{"role":"assistant","content":", 2"},"finish_reason":""}]}

...

data: {"id":"cmpl-c0","object":"chat.completion","created":1718000001,"model":"Distributed Model","choices":[{"index":0,"finish_reason":"stop"}]}

data: [DONE]
```
```

--------------------------------

### Convert Hugging Face Model

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_CONVERT_HF_MODEL.md

Use this script to convert a downloaded Hugging Face model. Ensure you are in the 'converter' directory and provide the path to your model and the desired quantization type and model name.

```sh
cd converter
python convert-hf.py path/to/hf/model q40 mistral-7b-0.3
```

--------------------------------

### Convert HuggingFace Model to Distributed Llama Format

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Converts downloaded HuggingFace models (safetensors format) to the Distributed Llama `.m` binary format. Supports multiple architectures and quantization options. Requires Python dependencies.

```sh
# Install Python dependencies
pip install safetensors torch

# 1. Convert model weights (produces dllama_model_mistral-7b-0.3_q40.m)
cd converter
python convert-hf.py /path/to/Mistral-7B-v0.3 q40 mistral-7b-0.3

# Float type options: f32, f16, q40 (4-bit quantized, recommended), q80

# 2. Convert tokenizer (produces dllama_tokenizer_mistral-7b-0.3.t)
python convert-tokenizer-hf.py /path/to/Mistral-7B-v0.3 mistral-7b-0.3

# 3. Run the converted model
cd ..
./dllama inference \
  --prompt "Hello world" \
  --steps 64 \
  --model dllama_model_mistral-7b-0.3_q40.m \
  --tokenizer dllama_tokenizer_mistral-7b-0.3.t \
  --buffer-float-type q80

# Llama3 variant (different tokenizer converter)
python convert-tokenizer-llama3.py /path/to/Meta-Llama-3-8B llama3-8b

```

--------------------------------

### Distributed Llama CLI Argument Reference

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Lists shared and specific arguments for `dllama` and `dllama-api` commands, including inference, API server, worker, and GPU options. Recommended buffer float type is q80.

```sh
# Inference / Chat / API shared arguments:
--model <path>               # Path to .m model file (required)
--tokenizer <path>           # Path to .t tokenizer file (required)
--buffer-float-type <type>   # Sync precision: f32 | f16 | q40 | q80 (q80 recommended)
--workers <ip:port> ...      # Space-separated list of worker addresses
--max-seq-len <n>            # Cap context length to reduce RAM usage (e.g. 4096)
--nthreads <n>               # CPU thread count (match physical core count)
--temperature <float>        # Sampling temperature, default 0.8
--topp <float>               # Top-p nucleus sampling threshold, default 0.9
--seed <int>                 # Random seed for reproducibility

# API server only:
--host <addr>                # Bind address (default 127.0.0.1; use 0.0.0.0 for LAN)
--port <port>                # HTTP port (default 9990)

# Worker / API only:
--host <addr>                # Bind address
--port <port>                # Listen port

# Inference only:
--prompt <text>              # Initial prompt string
--steps <n>                  # Number of tokens to generate

# GPU (Vulkan build only):
--gpu-index <n>              # GPU device index (0 = first GPU); requires --nthreads 1

```

--------------------------------

### GPU-Accelerated Inference

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Run inference on a single device using GPU acceleration.

```APIDOC
## ./dllama inference (GPU)

### Description
Performs inference on a single device with GPU acceleration.

### Method
Command-line execution

### Endpoint
N/A (Command-line tool)

### Parameters
- `--prompt` (string) - Required - The text prompt to generate a response from.
- `--steps` (integer) - Required - The number of generation steps.
- `--model` (string) - Required - Path to the model file.
- `--tokenizer` (string) - Required - Path to the tokenizer file.
- `--buffer-float-type` (string) - Required - The float type for buffers (e.g., q80).
- `--nthreads` (integer) - Required - Number of threads to use (typically 1 for single-thread GPU).
- `--gpu-index` (integer) - Required for GPU - The index of the GPU to use.

### Request Example
```sh
./dllama inference \
  --prompt "Hello world" \
  --steps 64 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 1 --gpu-index 0
```
```

--------------------------------

### Clone Repository and Compile Distributed Llama

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_LINUX_MACOS_WIN.md

Clones the Distributed Llama repository and compiles the main executable and API server.

```shell
git clone https://github.com/b4rtaz/distributed-llama.git
cd distributed-llama
make dllama
make dllama-api
```

--------------------------------

### Access API Server

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md

Access the running Distributed Llama API server from your computer using its IP address and port.

```bash
http://raspberrypi1.local:9999/v1/models
```

--------------------------------

### Run Distributed Llama Inference

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md

Execute an inference request on the root Raspberry Pi device, specifying the prompt, model, tokenizer, and worker addresses.

```bash
sudo nice -n -20 ./dllama inference \
  --prompt "Hello world" \
  --steps 32 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
```

--------------------------------

### Run Distributed Llama Inference

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_LINUX_MACOS_WIN.md

Executes inference on the root device, connecting to specified worker nodes. Ensure the model and tokenizer paths are correct.

```shell
./dllama inference \
  --prompt "Hello world" \
  --steps 32 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
```

--------------------------------

### Assign Static IP Addresses to Raspberry Pi Devices

Source: https://github.com/b4rtaz/distributed-llama/blob/main/docs/HOW_TO_RUN_RASPBERRYPI.md

Assign static IP addresses to each Raspberry Pi device on the 'eth0' interface. Ensure each IP is unique within the same subnet.

```bash
sudo ip addr add 10.0.0.1/24 dev eth0 # 🔸 ROOT
sudo ip addr add 10.0.0.2/24 dev eth0 # 🔹 WORKER 1
sudo ip addr add 10.0.0.3/24 dev eth0 # 🔹 WORKER 2
sudo ip addr add 10.0.0.4/24 dev eth0 # 🔹 WORKER 3
```

--------------------------------

### Verify Network Connectivity

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Check network connectivity from the root node to the worker nodes using the ping command.

```bash
ping 10.0.0.2
```

```bash
ping 10.0.0.3
```

```bash
ping 10.0.0.4
```

--------------------------------

### Non-streaming Chat Completion

Source: https://context7.com/b4rtaz/distributed-llama/llms.txt

Use this for standard chat completions where a single response is expected. Supports parameters like temperature, max_tokens, seed, and stop sequences.

```sh
curl -X POST http://localhost:9999/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user",   "content": "What is the capital of France?"}
    ],
    "temperature": 0.7,
    "max_tokens": 128,
    "seed": 42,
    "stop": ["<|eot_id|>"]
  }'
```