### Clone and Install Repository

Source: https://github.com/notpunchnox/rkllama/blob/main/converter/README.md

Initial setup steps to clone the repository and install necessary dependencies.

```bash
git clone https://github.com/yourusername/huggingface-to-rkllm.git
cd huggingface-to-rkllm
```

```bash
# Create virtual environment
python3 -m venv venv

# Activate virtual environment
# On macOS/Linux:
source venv/bin/activate
# On Windows:
.\venv\Scripts\activate
```

```bash
# Upgrade pip first
pip install --upgrade pip

# Install dependencies
pip install -r requirements.txt
```

--------------------------------

### Run the Setup Script

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/installation.md

Make the setup script executable and run it with root privileges. Use the --no-conda flag to skip Miniconda installation.

```bash
chmod +x setup.sh
sudo ./setup.sh
```

```bash
sudo ./setup.sh --no-conda
```

--------------------------------

### GET Request Example

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Example of how to make a GET request to the root endpoint.

```bash
curl -X GET http://localhost:8080/
```

--------------------------------

### Example Model Execution Command

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/Rebuild-arch.md

Example of running a specific model with its corresponding rkllm file and HuggingFace repository path.

```bash
rkllama run deepseek-llm-7b-chat-rk3588-w8a8-opt-0-hybrid-ratio-0.5 deepseek-llm-7b-chat-rk3588-w8a8-opt-0-hybrid-ratio-0.5.rkllm c01zaut/deepseek-llm-7b-chat-rk3588-1.1.1
```

--------------------------------

### Start RKLLama Server

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Use these commands to start the RKLLama server with default or custom settings, including debug mode and Docker deployment.

```bash
# Start the server with default settings (port 8080)
rkllama_server --models ~/RKLLAMA/models
```

```bash
# Start with debug mode for detailed logging
rkllama_server --debug --models ~/RKLLAMA/models
```

```bash
# Start with custom port
rkllama_server --port 9090 --models ~/RKLLAMA/models
```

```bash
# Docker deployment
docker run -it --privileged -p 8080:8080 \
  -v ~/RKLLAMA/models:/opt/rkllama/models \
  ghcr.io/notpunchnox/rkllama:main
```

--------------------------------

### Start the RKLLama Server

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/installation.md

Initialize the server process to begin serving models.

```bash
rkllama serve
```

--------------------------------

### Start RKLLama Client

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Initiate the RKLLama client. Use 'help' to see available commands.

```bash
rkllama_client
```

```bash
rkllama_client help
```

--------------------------------

### Example Conversion Commands

Source: https://github.com/notpunchnox/rkllama/blob/main/converter/README.md

Common command patterns for different model scenarios.

```bash
# Convert OPT-125M model with default settings
python3 converter.py facebook/opt-125m

# Convert Qwen2.5-7B with custom quantization
python3 converter.py Qwen/Qwen2.5-7B --quantization Q4_K_M

# Convert a private model using Hugging Face token
python3 converter.py your-org/private-model --token "hf_xxxxx"

# Convert model with CPU (when no GPU is available)
python3 converter.py Qwen/Qwen2.5-7B --device cpu
```

--------------------------------

### Initialized Modelfile Content

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/Rebuild-arch.md

Example of the generated Modelfile configuration values.

```env
FROM="deepseek-llm-7b-chat-rk3588-w8a8-opt-0-hybrid-ratio-0.5.rkllm"
HUGGINGFACE_PATH="c01zaut/deepseek-llm-7b-chat-rk3588-1.1.4"
SYSTEM=""
TEMPERATURE=1.0
```

--------------------------------

### Install RKLLama

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Install RKLLama using pip within a virtual environment. Ensure you have cloned the repository first.

```bash
python -m pip install .
```

--------------------------------

### POST /pull

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Downloads and installs a model from Hugging Face.

```APIDOC
## POST /pull

### Description
Downloads and installs a model from Hugging Face.

### Method
POST

### Endpoint
/pull

### Request Body
- **model** (string) - Required - The Hugging Face model path.

### Response
#### Success Response (200)
- **message** (string) - Download progress information.
```

--------------------------------

### Multiple Tool Calls

Source: https://context7.com/notpunchnox/rkllama/llms.txt

This example shows how to define multiple tools (weather and calculator) and have the model determine which to call based on the user's request.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "user", "content": "Get the weather in Tokyo and calculate 25 * 4"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get weather information",
          "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"]
          }
        }
      },
      {
        "type": "function",
        "function": {
          "name": "calculator",
          "description": "Perform mathematical calculations",
          "parameters": {
            "type": "object",
            "properties": {"expression": {"type": "string"}},
            "required": ["expression"]
          }
        }
      }
    ]
  }'
```

--------------------------------

### Start RKLLama with Docker Compose

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Use Docker Compose to easily set up and run RKLLama, managing configurations like volumes automatically.

```bash
docker compose up --detach --remove-orphans
```

--------------------------------

### Tool Call Example with cURL

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/ollama-compatibility.md

Demonstrates making a tool call for function execution. This requires defining tools with their functions, descriptions, and parameters.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "user", "content": "Get the weather in Tokyo"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather for a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "City name"
              }
            },
            "required": ["location"]
          }
        }
      }
    ]'
  }'
```

--------------------------------

### Run a Model with RKLLama Client

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Start an interactive chat session with a specified model using the RKLLama client.

```bash
rkllama_client run <model_name>
```

--------------------------------

### GET / - Welcome Message

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

This endpoint displays a welcome message and provides a link to the project's GitHub repository.

```APIDOC
## GET /

### Description
Displays a welcome message and a link to the GitHub project.

### Method
GET

### Endpoint
/

### Response
#### Success Response (200)
- **message** (string) - A welcome message.
- **github** (string) - A URL to the GitHub repository.

#### Response Example
```json
{
  "message": "Welcome to RK-LLama!",
  "github": "https://github.com/notpunhnox/rkllama"
}
```

### Request Example
```bash
curl -X GET http://localhost:8080/
```
```

--------------------------------

### Start RKLLama Server

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Launch the RKLLama server, specifying the directory where models are stored. Debug mode can be enabled with the --debug flag.

```bash
rkllama_server --models <models_dir>
```

```bash
rkllama_server --debug --models <models_dir>
```

--------------------------------

### Download Model from Hugging Face

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Downloads and installs a model from Hugging Face. Specify the model in the format 'hf/username/repo/file.rkllm'.

```http
POST /pull
Content-Type: application/json
```

```json
{
  "model": "hf/username/repo/file.rkllm"
}
```

```txt
Downloading <file> (<size> MB)...
<progress>%
```

```txt
Error during download: <error>
```

```bash
curl -X POST http://localhost:8080/pull \
-H "Content-Type: application/json" \
-d '{"model": "hf/username/repo/file.rkllm"}'
```

--------------------------------

### Tool Calling Example with cURL

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Demonstrates how to use cURL to make a POST request to the RKLLama API for tool/function calling, specifically for fetching weather information.

```bash
# Example: Weather tool call
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "City name"}}, "required": ["location"]}}}
    ]}'
```

--------------------------------

### GET /models

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Returns the list of available models in the ~/RKLLAMA/models directory.

```APIDOC
## GET /models

### Description
Returns the list of available models in the ~/RKLLAMA/models directory.

### Method
GET

### Endpoint
/models

### Response
#### Success Response (200)
- **models** (array) - List of available model filenames.

#### Response Example
{
  "models": [
    "model1.rkllm",
    "model2.rkllm",
    "model3.rkllm"
  ]
}
```

--------------------------------

### POST /pull and /api/pull

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Downloads and installs models from Hugging Face.

```APIDOC
## POST /pull

### Description
Downloads a model from a Hugging Face repository.

### Method
POST

### Endpoint
http://localhost:8080/pull

### Request Body
- **model** (string) - Required - The Hugging Face model path
- **model_name** (string) - Required - The local name to assign to the model
```

--------------------------------

### View Debug Logs

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Example debug logs showing tool searching and detection. Enable debug mode for detailed logs when troubleshooting.

```log
2024-12-21 10:30:00 - rkllama.format_utils - DEBUG - Searching tools with standard method: get_tool_calls_standard
2024-12-21 10:30:00 - rkllama.format_utils - DEBUG - Searching tools with generic method: get_tool_calls_generic
2024-12-21 10:30:00 - rkllama.server_utils - DEBUG - Tool calls detected: [{'function': {'name': 'get_weather', 'arguments': {'location': 'Paris'}}}]
```

--------------------------------

### RKLLAMA Configuration API - Basic Usage

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/configuration.md

Demonstrates basic usage of the RKLLAMA configuration API for getting and setting values, including type inference and explicit type casting.

```APIDOC
## RKLLAMA Configuration API - Basic Usage

### Description
This section covers the fundamental operations of the RKLLAMA configuration API, including retrieving configuration values with automatic type inference or explicit type casting, and setting new configuration values.

### Method
N/A (Python API calls)

### Endpoint
N/A (Python API calls)

### Parameters
N/A

### Request Example
```python
from rkllama import config

# Get values with automatic type inference
port = config.get("server", "port")
debug = config.get("server", "debug")

# Get values with explicit types
port = config.get("server", "port", 8080, as_type=int)
debug = config.get("server", "debug", False, as_type=bool)

# Get path values (resolved against app_root)
models_dir = config.get_path("models")

# Set values
config.set("server", "port", 9090)
config.set("logging", "level", "DEBUG")
```

### Response
#### Success Response (200)
N/A (Python API calls return values directly)

#### Response Example
```python
# Example return values from config.get()
# port might be 8080 (int)
# debug might be False (bool)
# models_dir might be "/path/to/app/models" (str)
```
```

--------------------------------

### RKLLama Server Configuration (INI)

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Configure RKLLama server settings using an INI file. This example shows how to set the port, debug mode, and model/data paths. Environment variables can override these settings.

```ini
# ~/.config/rkllama/rkllama.ini

[server]
port = 8080
debug = false

[paths]
models = ~/RKLLAMA/models
data = ~/RKLLAMA/data
logs = ~/RKLLAMA/logs

[model]
default_temperature = 1.0
default_num_ctx = 4096
default_max_new_tokens = 2048
default_top_k = 40
default_top_p = 0.9
default_repeat_penalty = 1.1
default_frequency_penalty = 0.0
default_presence_penalty = 0.0
default_enable_thinking = false
```

--------------------------------

### Tool Calling for Weather Information

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Utilize the tool calling feature to enable the model to invoke external functions. This example demonstrates requesting weather information.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              },
              "format": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"],
                "description": "Temperature unit to use"
              }
            },
            "required": ["location"]
          }
        }
      }
    ]
  }'
```

--------------------------------

### Call Multiple Tools with RKLLama

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Example of using curl to request both weather and time information for Paris by defining two separate tools.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "user", "content": "Get the weather in Paris and current time there"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get weather information",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {"type": "string"}
            },
            "required": ["location"]
          }
        }
      },
      {
        "type": "function",
        "function": {
          "name": "get_time",
          "description": "Get current time in a timezone",
          "parameters": {
            "type": "object",
            "properties": {
              "timezone": {"type": "string"}
            },
            "required": ["timezone"]
          }
        }
      }
    ]'

```

--------------------------------

### List Available Models with cURL

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/ollama-compatibility.md

Retrieve a list of all available models on the RKLLAMA server using a simple GET request.

```bash
curl http://localhost:8080/api/tags
```

--------------------------------

### Enable Debug Mode for RKLLAMA

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/ollama-compatibility.md

Start the RKLLAMA server with the `--debug` flag to enable optional debugging features for troubleshooting.

```bash
rkllama serve --debug
```

--------------------------------

### OpenAI-Compatible Chat Completion

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Interact using the OpenAI-compatible chat completions endpoint for broader client compatibility. This example shows a basic request with system and user messages.

```bash
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "system", "content": "You are a coding assistant."},
      {"role": "user", "content": "Write a hello world in Rust"}
    ],
    "temperature": 0.7,
    "max_tokens": 512,
    "stream": false
  }'
```

--------------------------------

### Generate Speech with MMS-TTS

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Use MMS-TTS models for text-to-speech conversion. This example demonstrates generating speech in Spanish and saving it as a WAV file.

```bash
curl -X POST http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mms_tts_spa",
    "input": "Este es un ejemplo de texto a voz.",
    "response_format": "wav"
  }' --output output.wav
```

--------------------------------

### List Available Models using cURL

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/model_naming.md

To view all available models and their simplified names, send a GET request to the /v1/models endpoint using cURL.

```bash
curl http://localhost:8080/v1/models
```

--------------------------------

### Pull Model from Hugging Face via REST API

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Download and install models from Hugging Face repositories using the REST API. This method requires specifying the model file and a local name.

```bash
curl -X POST http://localhost:8080/pull \
  -H "Content-Type: application/json" \
  -d '{
    "model": "c01zaut/Qwen2.5-3B-Instruct-RK3588-1.1.4/Qwen2.5-3B-Instruct-rk3588-w8a8.rkllm",
    "model_name": "qwen2.5:3b"
  }'
```

--------------------------------

### Call Weather Tool with RKLLama

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Example of using curl to send a request to RKLLama for weather information in Tokyo, specifying the 'get_current_weather' tool.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "user", "content": "What is the weather like in Tokyo?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather in a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city name"
              },
              "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"],
                "description": "Temperature unit"
              }
            },
            "required": ["location"]
          }
        }
      }
    ]'

```

--------------------------------

### Configure Modelfile for Multimodal Encoder (.rknn) Models

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

When using multimodal encoder models (.rknn), include these properties in your Modelfile to define image dimensions, token counts, and special tokens for image handling. Examples for Qwen2VL and MiniCPMV4 are provided.

```env
IMAGE_WIDTH=448
IMAGE_HEIGHT=
N_IMAGE_TOKENS=
IMG_START=
IMG_END=
IMG_CONTENT=
```

```env
# For example, for Qwen2VL/Qwen2.5VL:

IMAGE_WIDTH=392
IMAGE_HEIGHT=392
N_IMAGE_TOKENS=196
IMG_START=<|vision_start|>
IMG_END=<|vision_end|>
IMG_CONTENT=<|image_pad|>
```

```env
# For example, for MiniCPMV4:

IMAGE_WIDTH=448
IMAGE_HEIGHT=448
N_IMAGE_TOKENS=64
IMG_START=<image>
IMG_END=</image>
IMG_CONTENT=<unk>
```

--------------------------------

### Call Calculator Tool with RKLLama

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Example of using curl to send a request to RKLLama for a mathematical calculation, specifying the 'calculator' tool with an expression.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "user", "content": "Calculate 25 * 4 + 10"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "calculator",
          "description": "Perform mathematical calculations",
          "parameters": {
            "type": "object",
            "properties": {
              "expression": {
                "type": "string",
                "description": "Mathematical expression to evaluate"
              }
            },
            "required": ["expression"]
          }
        }
      }
    ]'

```

--------------------------------

### Node.js Client for Tool Calling

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Use the Node.js client to call the RKLLama API with tools. Ensure the 'axios' library is installed. This async function returns detected tool calls or null, logging errors.

```javascript
const axios = require('axios');

async function callWithTools(prompt, tools) {
    try {
        const response = await axios.post('http://localhost:8080/api/chat', {
            model: 'qwen2.5:3b',
            messages: [{role: 'user', content: prompt}],
            tools: tools
        });
        
        if (response.data.message?.tool_calls) {
            return response.data.message.tool_calls;
        }
        return null;
    } catch (error) {
        console.error('Error:', error.response?.data || error.message);
        return null;
    }
}

// Example usage
const weatherTool = {
    type: "function",
    function: {
        name: "get_weather",
        description: "Get weather information",
        parameters: {
            type: "object",
            properties: {
                location: {type: "string"}
            },
            required: ["location"]
        }
    }
};

callWithTools("What's the weather in Tokyo?", [weatherTool])
    .then(tools => console.log('Tools called:', tools));
```

--------------------------------

### GET /current_model

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Returns the name of the currently loaded model.

```APIDOC
## GET /current_model

### Description
Returns the name of the currently loaded model.

### Method
GET

### Endpoint
/current_model

### Response
#### Success Response (200)
- **model_name** (string) - The name of the loaded model.

#### Response Example
{
  "model_name": "model_name"
}
```

--------------------------------

### Launch Model and Generate Modelfile

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/Rebuild-arch.md

Command to launch a model and initialize the Modelfile using a HuggingFace repository for tokenizer and template configuration.

```bash
rkllama run modelname file.rkllm huggingface_repo
```

--------------------------------

### Troubleshooting and Configuration

Source: https://github.com/notpunchnox/rkllama/blob/main/converter/README.md

Commands for resolving common dependency or environment issues.

```bash
# Install PyTorch separately
pip install torch torchvision torchaudio
```

```bash
export HF_TOKEN="your_token_here"
```

```bash
python3 converter.py --verbose Qwen/Qwen2.5-7B
```

--------------------------------

### Get Current Loaded Model

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Retrieves the name of the model that is currently loaded in memory.

```http
GET /current_model
```

```json
{
  "model_name": "model_name"
}
```

```json
{
  "error": "No model is currently loaded."
}
```

```bash
curl -X GET http://localhost:8080/current_model
```

--------------------------------

### Welcome Message Endpoint

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

The root endpoint '/' returns a welcome message and a link to the GitHub project.

```json
{
  "message": "Welcome to RK-LLama!",
  "github": "https://github.com/notpunhnox/rkllama"
}
```

--------------------------------

### RKLLAMA Configuration API - Displaying and Saving

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/configuration.md

Shows how to display the current configuration, validate it, and save it to a project configuration file.

```APIDOC
## RKLLAMA Configuration API - Displaying and Saving

### Description
This section details how to interact with the RKLLAMA configuration system to view the current settings, ensure their validity, and persist them to disk.

### Method
N/A (Python API calls)

### Endpoint
N/A (Python API calls)

### Parameters
N/A

### Request Example
```python
from rkllama import config

# Display current configuration
config.display()

# Validate configuration
if not config.validate():
    print("Configuration validation failed!")

# Save to project configuration file
config.save_to_project_ini()
```

### Response
#### Success Response (200)
N/A (Python API calls perform actions or print output)

#### Response Example
```python
# config.display() will print the configuration to stdout.
# config.validate() returns True if valid, False otherwise.
# config.save_to_project_ini() saves the configuration to a file.
```
```

--------------------------------

### Access the RKLLama Client

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/installation.md

Launch the client interface to interact with the system.

```bash
rkllama
```

--------------------------------

### List Available Models with RKLLama Client

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

View all available models managed by the RKLLama client.

```bash
rkllama_client list
```

--------------------------------

### RKLLama Error Response for Missing Parameters

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Example of a JSON error response from RKLLama indicating a missing required parameter for a tool call.

```json
{
  "error": "Missing required field: location"
}

```

--------------------------------

### Docker Conversion Workflow

Source: https://github.com/notpunchnox/rkllama/blob/main/converter/README.md

Building and running the converter within a Docker container.

```bash
docker build -t huggingface-to-rkllm .
```

```bash
docker run -v /path/to/models:/models huggingface-to-rkllm convert \
    --model-id Qwen/Qwen2.5-7B \
    --output-dir /models/output \
    --quantization Q4_0
```

--------------------------------

### Execute a Basic Tool Call Request

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Use a POST request to the chat API to initiate a tool call with defined function parameters.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              },
              "format": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"],
                "description": "The temperature unit to use"
              }
            },
            "required": ["location"]
          }
        }
      }
    ]
  }'
```

--------------------------------

### Execute RKLLAMA via Command-line

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/configuration.md

Commands for specifying configuration files or overriding settings directly via arguments.

```bash
# Load specific configuration file
python -m rkllama --config /path/to/config.ini

# Set server section values
python -m rkllama --server_port 8080 --server_debug
```

--------------------------------

### View New Directory Structure

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/Rebuild-arch.md

Visual representation of the required directory layout for models.

```text
~/RKLLAMA
    └── models
        |
        |── DeepSeek-v3-7b
        |   |── Modelfile
        |   └── deepseek.rkllm
        |
        └── Llama3-7b
            |── Modelfile
            └── deepseek.rkllm
```

--------------------------------

### RKLLAMA Configuration API - Reloading

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/configuration.md

Explains how to reload the configuration from all sources after changes have been made.

```APIDOC
## RKLLAMA Configuration API - Reloading

### Description
This section describes the process of reloading the RKLLAMA configuration from all defined sources, ensuring that any external changes to configuration files or environment variables are reflected in the application.

### Method
N/A (Python API calls)

### Endpoint
N/A (Python API calls)

### Parameters
N/A

### Request Example
```python
from rkllama import config

# Reload configuration from all sources
config.reload_config()
```

### Response
#### Success Response (200)
N/A (Python API calls perform actions)

#### Response Example
```python
# config.reload_config() updates the internal configuration state.
```
```

--------------------------------

### Clone the RKLLama Repository

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/installation.md

Download the source code from the official repository and navigate into the directory.

```bash
git clone https://github.com/notpunchnox/rkllama
cd rkllama
```

--------------------------------

### Python Client for Tool Calling

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Use the Python client to call the RKLLama API with tools. Ensure the 'requests' library is installed. This function returns detected tool calls or None.

```python
import requests

def call_with_tools(prompt, tools):
    response = requests.post('http://localhost:8080/api/chat', json={
        'model': 'qwen2.5:3b',
        'messages': [{'role': 'user', 'content': prompt}],
        'tools': tools
    })
    
    data = response.json()
    if 'tool_calls' in data.get('message', {}):
        return data['message']['tool_calls']
    return None

# Example usage
weather_tool = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather information",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"}
            },
            "required": ["location"]
        }
    }
}

tools_called = call_with_tools("What's the weather in Tokyo?", [weather_tool])
if tools_called:
    print("Tools called:", tools_called)
```

--------------------------------

### Command Line Interface Usage

Source: https://github.com/notpunchnox/rkllama/blob/main/converter/README.md

Basic and advanced usage of the converter CLI.

```bash
# Basic conversion
python3 converter.py Qwen/Qwen2.5-7B

# Conversion with custom options
python3 converter.py Qwen/Qwen2.5-7B \
    --output-dir "models/converted" \
    --quantization "Q4_K_M" \
    --max-context-len 8192 \
    --dtype "float16" \
    --device "cuda" \
    --token "your_hf_token"
```

--------------------------------

### Download Model with RKLLama Client

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Use the `rkllama_client pull` command to download models from Hugging Face. You can specify the full path or run interactively.

```bash
rkllama_client pull username/repo_id/model_file.rkllm/custom_model_name
```

```bash
rkllama_client pull
Repo ID ( example: punchnox/Tinnyllama-1.1B-rk3588-rkllm-1.1.4): <your response>
File ( example: TinyLlama-1.1B-Chat-v1.0-rk3588-w8a8-opt-0-hybrid-ratio-0.5.rkllm): <your response>
Custom Model Name ( example: tinyllama-chat:1.1b ): <your response>
```

--------------------------------

### Configure Modelfile for .rkllm Models

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Use this configuration in your Modelfile when using .rkllm models. Specify the model file, Hugging Face path for tokenizer/chattemplate, and system prompt.

```env
FROM="file.rkllm"
HUGGINGFACE_PATH="huggingface_repository"
SYSTEM="Your system prompt"
TEMPERATURE=1.0
```

--------------------------------

### Basic Text Generation

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Use this endpoint for simple text generation tasks. Specify the model and prompt to receive a text response.

```bash
curl -X POST http://localhost:8080/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "prompt": "Write a Python function to calculate Fibonacci numbers"
  }'
```

--------------------------------

### Manage Configuration State

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/configuration.md

Utilities for displaying, validating, and persisting configuration changes.

```python
# Display current configuration
config.display()

# Validate configuration
if not config.validate():
    print("Configuration validation failed!")

# Save to project configuration file
config.save_to_project_ini()
```

--------------------------------

### Stream Tool Calls

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Enable streaming by setting the stream parameter to true in the request body.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [{"role": "user", "content": "Get weather for NYC"}],
    "tools": [/* tool definitions */],
    "stream": true
  }'
```

--------------------------------

### List Available Models

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Use this endpoint to retrieve a list of all models available in the RKLLAMA models directory. Ensure the models directory exists.

```http
GET /models
```

```json
{
  "models": [
    "model1.rkllm",
    "model2.rkllm",
    "model3.rkllm"
  ]
}
```

```json
{
  "error": "The directory ~/RKLLAMA/models is not found."
}
```

```bash
curl -X GET http://localhost:8080/models
```

--------------------------------

### Text Generation with Parameters

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Customize generation behavior using parameters like temperature, top_p, and top_k. Set 'stream' to false for a single response.

```bash
curl -X POST http://localhost:8080/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "prompt": "Creative story about a robot:",
    "options": {
      "temperature": 0.8,
      "top_p": 0.9,
      "top_k": 40,
      "num_predict": 256
    },
    "stream": false
  }'
```

--------------------------------

### Project Directory Structure

Source: https://github.com/notpunchnox/rkllama/blob/main/converter/README.md

Visual representation of the project's file and folder hierarchy.

```text
converter/
├── src/
│   ├── converter.py      # Main conversion module
│   ├── quantization.py   # Quantization handling
│   └── utils.py         # Utility functions
├── tests/
│   └── test_converter.py # Unit tests
├── requirements.txt     # Python dependencies
└── README.md           # Documentation
```

--------------------------------

### Reload Configuration

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/configuration.md

Refresh the configuration state from all defined sources.

```python
# Reload configuration from all sources
config.reload_config()
```

--------------------------------

### Basic Tool Call Request

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Send a POST request to the /api/chat endpoint with messages and tool definitions to initiate a tool call.

```bash
curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:3b",
    "messages": [
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              },
              "format": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"],
                "description": "Temperature unit to use"
              }
            },
            "required": ["location"]
          }
        }
      }
    ] 
  }'
```

--------------------------------

### POST /v1/audio/speech

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Converts text input into speech audio.

```APIDOC
## POST /v1/audio/speech

### Description
Generates speech audio from text using Piper TTS or MMS-TTS models.

### Method
POST

### Endpoint
http://localhost:8080/v1/audio/speech

### Request Body
- **model** (string) - Required - The TTS model name
- **input** (string) - Required - The text to convert to speech
- **voice** (string) - Optional - Voice identifier
- **response_format** (string) - Optional - Output format (e.g., wav)
```

--------------------------------

### RKLLama Modelfile Configuration

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Define model-specific settings using a Modelfile. This includes base model, tokenizer path, system prompt, and generation parameters like temperature and context size. Supports vision model parameters.

```bash
# ~/RKLLAMA/models/qwen2.5:3b/Modelfile

FROM="Qwen2.5-3B-Instruct-rk3588-w8a8.rkllm"

HUGGINGFACE_PATH="Qwen/Qwen2.5-3B-Instruct"

SYSTEM="You are a helpful AI assistant."

TEMPERATURE=0.7

NUM_CTX=4096

MAX_NEW_TOKENS=2048

TOP_K=40

TOP_P=0.9

REPEAT_PENALTY=1.1

FREQUENCY_PENALTY=0.0

PRESENCE_PENALTY=0.0

ENABLE_THINKING=false

# Vision model example (Qwen2VL)
# IMAGE_WIDTH=392
# IMAGE_HEIGHT=392
# N_IMAGE_TOKENS=196
# IMG_START=<|vision_start|>
# IMG_END=<|vision_end|>
# IMG_CONTENT=<|image_pad|>
```

--------------------------------

### Access Configuration via Python API

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/configuration.md

Methods for retrieving, setting, and managing configuration values programmatically.

```python
from rkllama import config

# Get values with automatic type inference
port = config.get("server", "port")
debug = config.get("server", "debug")

# Get values with explicit types
port = config.get("server", "port", 8080, as_type=int)
debug = config.get("server", "debug", False, as_type=bool)

# Get path values (resolved against app_root)
models_dir = config.get_path("models")

# Set values
config.set("server", "port", 9090)
config.set("logging", "level", "DEBUG")
```

--------------------------------

### Load a Model

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/english.md

Loads a specified model into memory. A model must be loaded before generating output. Ensure the model exists in the models directory.

```http
POST /load_model
Content-Type: application/json
```

```json
{
  "model_name": "model_name.rkllm"
}
```

```json
{
  "message": "Model <model_name> loaded successfully."
}
```

```json
{
  "error": "A model is already loaded. Please unload it first."
}
```

```json
{
  "error": "Model <model_name> not found in the /models directory."
}
```

```bash
curl -X POST http://localhost:8080/load_model \
-H "Content-Type: application/json" \
-d '{"model_name": "model1.rkllm"}'
```

--------------------------------

### List Available Models via REST API

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Retrieve all models available in the server's models directory using the REST API or the Ollama-compatible endpoint.

```bash
# Using REST API
curl -X GET http://localhost:8080/models

# Response:
# {
#   "models": [
#     "qwen2.5:3b",
#     "llama3-instruct:8b",
#     "deepseek-coder:7b"
#   ]
# }
```

```bash
# Ollama-compatible endpoint
curl http://localhost:8080/api/tags

# Response:
# {
#   "models": [
#     {
#       "name": "qwen2.5:3b",
#       "model": "qwen2.5:3b",
#       "modified_at": "2024-12-21T10:00:00Z",
#       "size": 3221225472,
#       "details": {
#         "family": "qwen2.5",
#         "parameter_size": "3B",
#         "quantization_level": "w8a8"
#       }
#     }
#   ]
# }
```

--------------------------------

### Run RKLLama Docker Container

Source: https://github.com/notpunchnox/rkllama/blob/main/README.md

Execute the RKLLama Docker container, mapping the host models directory to the container and exposing the server port.

```bash
docker run -it --privileged -p 8080:8080 -v <local_models_dir>:/opt/rkllama/models ghcr.io/notpunchnox/rkllama:main
```

--------------------------------

### Define Function Schema

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/tools.md

Tools must be defined using a specific JSON structure including name, description, and parameter properties.

```json
{
  "type": "function",
  "function": {
    "name": "function_name",
    "description": "Clear description of what the function does",
    "parameters": {
      "type": "object",
      "properties": {
        "param1": {
          "type": "string|number|boolean|array|object",
          "description": "Parameter description",
          "enum": ["option1", "option2"]  // Optional for string types
        },
        "param2": {
          "type": "number",
          "description": "Another parameter"
        }
      },
      "required": ["param1"]  // List of required parameters
    }
  }
}
```

--------------------------------

### RKLLama Client Commands

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Interact with the RKLLama server using the CLI client for model management and chat sessions. Ensure the server is running before using client commands.

```bash
# List available models
rkllama_client list
```

```bash
# Check running models in memory
rkllama_client ps
```

```bash
# Load a model into memory
rkllama_client load qwen2.5:3b
```

```bash
# Unload a model from memory
rkllama_client unload qwen2.5:3b
```

```bash
# Start interactive chat session
rkllama_client run qwen2.5:3b
```

```bash
# Pull a model from Hugging Face
rkllama_client pull punchnox/Qwen2.5-3B-rk3588/Qwen2.5-3B-Instruct.rkllm/qwen2.5:3b
```

```bash
# Show model information
rkllama_client info qwen2.5:3b
```

```bash
# Delete a model
rkllama_client rm qwen2.5:3b
```

```bash
# Check for updates
rkllama_client update
```

--------------------------------

### Show Model Information

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Retrieve detailed information about a specific model using its name. The response includes details like modification time, size, and parameters.

```bash
curl -X POST http://localhost:8080/api/show \
  -H "Content-Type: application/json" \
  -d '{"name": "qwen2.5:3b"}'
```

--------------------------------

### RKLLAMA Command-line Arguments

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/configuration.md

Details how to use command-line arguments to override RKLLAMA configuration settings, including specifying a configuration file and setting specific values.

```APIDOC
## RKLLAMA Command-line Arguments

### Description
RKLLAMA supports command-line arguments for overriding configuration settings with the highest priority. This allows for dynamic configuration adjustments without modifying configuration files.

### Method
N/A (Command-line execution)

### Endpoint
N/A (Command-line execution)

### Parameters
#### Command-line Arguments
- `--config` (string) - Optional - Path to a specific configuration file to load.
- `--server_port` (integer) - Optional - Overrides the `port` setting in the `server` section.
- `--server_debug` (boolean) - Optional - Overrides the `debug` setting in the `server` section (enables debug mode).

### Request Example
```bash
# Load specific configuration file
python -m rkllama --config /path/to/config.ini

# Set server section values
python -m rkllama --server_port 8080 --server_debug
```

### Response
#### Success Response (200)
N/A (Command-line execution)

#### Response Example
N/A
```

--------------------------------

### Run Project Tests

Source: https://github.com/notpunchnox/rkllama/blob/main/converter/README.md

Command to execute the unit tests using pytest.

```bash
pytest tests/
```

--------------------------------

### Enable RKLLAMA Server Debug Mode

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/ollama-compatibility.md

Run the server script with the --debug flag to enable detailed logging to `rkllama_debug.log` and additional console output. This mode is optional and not required for normal operation.

```bash
bash ~/RKLLAMA/server.sh --debug
```

--------------------------------

### View Previous Directory Structure

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/Rebuild-arch.md

Visual representation of the directory layout prior to version 0.0.3.

```text
~/RKLLAMA
    └── models
        |── llama3-7b.rkllm
        └── qwen2.5-3b.rkllm
```

--------------------------------

### Server Log Output

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/Rebuild-arch.md

Expected output in server logs after initiating a model run.

```bash
FROM: deepseek-llm-7b-chat-rk3588-w8a8-opt-0-hybrid-ratio-0.5.rkllm
HuggingFace Path: c01zaut/deepseek-llm-7b-chat-rk3588-1.1.1
```

--------------------------------

### Generate Completion with cURL

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/api/ollama-compatibility.md

This endpoint is for single-turn completions based on a given prompt. Specify the model and the prompt text.

```bash
curl -X POST http://localhost:8080/api/generate -d '{
  "model": "qwen2.5:3b",
  "prompt": "Write a poem about AI"
}'
```

--------------------------------

### Reorganize Model Directories

Source: https://github.com/notpunchnox/rkllama/blob/main/documentation/Guide/en/Rebuild-arch.md

Command to automatically migrate existing model files into the new folder-based structure.

```bash
rkllama list
```

--------------------------------

### POST /api/show

Source: https://context7.com/notpunchnox/rkllama/llms.txt

Retrieves detailed information about a specific model.

```APIDOC
## POST /api/show

### Description
Returns metadata, configuration, and details for a specific model.

### Method
POST

### Endpoint
http://localhost:8080/api/show

### Request Body
- **name** (string) - Required - The name of the model
```