FastFlowLM

https://fastflowlm.com/docs
Admin
FastFlowLM is an NPU-first LLM inference runtime delivering ultra-efficient language model execution...
Tokens:21,894
Snippets:324
Trust Score:5.7
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://fastflowlm.com/docs/instructions/server/websearch_LangChain

Install necessary LangChain and utility packages.

```bash
pip install -U langchain langchain-community langchainhub langchain-ollama langchain-huggingface sentence-transformers ddgs beautifulsoup4 requests ollama
```

--------------------------------

### Verify Installation with CLI

Source: https://fastflowlm.com/docs/install_win

Run a model in the terminal to verify the installation.

```bash
flm run llama3.2:1b
```

--------------------------------

### Setup Virtual Environment

Source: https://fastflowlm.com/docs/instructions/server/websearch_LangChain

Commands to create and activate a Python virtual environment.

```bash
python -m venv rag_websearch-env
.\rag_websearch-env\Scripts\activate
```

--------------------------------

### Install XRT and NPU Drivers on Ubuntu

Source: https://fastflowlm.com/docs/install_lin

Installs the necessary XRT and NPU driver packages.

```bash
sudo apt install libxrt-npu2 amdxdna-dkms

```

--------------------------------

### Example Hotkey Assignments

Source: https://fastflowlm.com/docs/instructions/server/obsidian

Suggested hotkey mappings for Local GPT plugin actions.

```text
Alt + H → General Help
Alt + C → Continue Writing
Alt + S → Summarize
Alt + G → Fix Spelling and Grammar
Alt + F → Find Action Items
```

--------------------------------

### Install FastFlowLM Package

Source: https://fastflowlm.com/docs/install_lin

Installs the downloaded .deb package.

```bash
sudo apt install ./fastflowlm*.deb

```

--------------------------------

### Install Python Dependencies

Source: https://fastflowlm.com/docs/instructions/server/rag_LangChain

Installs necessary libraries for LangChain, FAISS, and local LLM integration.

```bash
pip install -U langchain langchain-community langchain-huggingface sentence-transformers faiss-cpu tiktoken ollama langchain-ollama
```

--------------------------------

### Create Project Directory

Source: https://fastflowlm.com/docs/instructions/server/webui

Command to create and enter a new directory for the Docker setup.

```bash
mkdir open-webui && cd open-webui
```

--------------------------------

### Build FastFlowLM from Source

Source: https://fastflowlm.com/docs/install_lin

Compiles and installs the project using CMake presets.

```bash
cd src
cmake --preset linux-default
cmake --build --preset linux-default -j$(nproc)
cmake --install --preset linux-default

```

--------------------------------

### List Models

Source: https://fastflowlm.com/docs/instructions/cli

Filter and display available models based on their installation status.

```bash
flm list --filter all
flm list --filter installed
flm list --filter not-installed
```

--------------------------------

### Initialize RAG Script Imports

Source: https://fastflowlm.com/docs/instructions/server/websearch_LangChain

Initial imports and setup for the websearch_rag.py script.

```python
# websearch_rag.py

import warnings
import os
from ddgs import DDGS
from langchain_ollama import OllamaLLM
from langchain.prompts import PromptTemplate
from langchain.chains import RetrievalQA

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document

warnings.filterwarnings("ignore", category=ResourceWarning)
```

--------------------------------

### Example Playground Prompt

Source: https://fastflowlm.com/docs/instructions/server/msft_AI_toolkit

Sample input text for testing model inference in the AI Toolkit Playground.

```text
What are the benefits of local inference?
```

--------------------------------

### Implement get_temperature tool calling in Python

Source: https://fastflowlm.com/docs/instructions/server/tool_calling

A complete example showing how to define a tool, stream a request to the FastFlowLM server, process tool calls, and provide the tool output back to the model.

```python
from openai import OpenAI
import json

# Connect to local FastFlowLM server
client = OpenAI(
    base_url="http://127.0.0.1:52625/v1",
    api_key="flm",
)

# OpenAI Chat Completions Tool definition
tools = [
{
    "type": "function",
    "function": {
            "name": "get_temperature",
            "description": "Get the current temperature for a city",
            "parameters": {
                "type": "object",
                "required": ["city"],
                "properties": {
                    "city": {"type": "string", "description": "The name of the city"}
                }
            }
        }
    }
]

# Tool implementation
def get_temperature(city: str) -> str:
    return {"New York":"22°C","London":"15°C","Tokyo":"18°C"}.get(city, "Unknown")

# Step 1:
## Original user message
messages = [
    {"role": "user", "content": "What is the temperature in New York?"}
]

## Ask model; it should emit a tool call
tool_calls = []
assistant_content = ""
for chunk in client.chat.completions.create(
    model="qwen3-it:4b",
    messages=messages,
    tools=tools,
    stream=True,
):
    delta = chunk.choices[0].delta

    # Collect assistant text (if any)
    if hasattr(delta, "content") and delta.content is not None:
        assistant_content += delta.content

    # Collect tool call tokens
    if hasattr(delta, "tool_calls") and delta.tool_calls is not None:
        for tc in delta.tool_calls:
            tool_calls.append(tc)


# Step 2:
## Add assistant message (with tool calls) to messages
messages.append(
    {
        "role": "assistant",
        "content": assistant_content,      # may be None
        "tool_calls": tool_calls # important: keep tool_calls
    }
)

## Execute tool calls and add tool call outputs back
if tool_calls:
    for tc in tool_calls:
        if tc.function.name == "get_temperature":
            args = json.loads(tc.function.arguments)
            temp = get_temperature(args["city"])
            messages.append(
                {
                    "role": "tool",
                    "tool_call_id": tc.id,
                    "content": json.dumps({"temperature": temp}),
                }
            )

print("Messages so far:")
print(messages)

# Step 3:
## Combine all the messages and ask again
for chunk in client.chat.completions.create(
    model="qwen3-it:4b",
    messages=messages,
    tools=tools,
    stream=True,
):
    delta = chunk.choices[0].delta
    if hasattr(delta, "content") and delta.content is not None:
        print(delta.content, end="", flush=True)
```

--------------------------------

### Launch FastFlowLM Server

Source: https://fastflowlm.com/docs/instructions/server/rag_LangChain

Starts the local FastFlowLM server to expose the REST API.

```bash
flm serve llama3.2:3b
```

--------------------------------

### Install OpenAI SDK

Source: https://fastflowlm.com/docs/instructions/server/openapi

Updates the OpenAI Python library to ensure compatibility with the API.

```bash
pip install --upgrade openai
```

--------------------------------

### Manage Docker Containers

Source: https://fastflowlm.com/docs/instructions/server/webui

Commands for starting, stopping, and maintaining the Open WebUI container.

```bash
docker compose up -d
```

```bash
docker logs -f open-webui
```

```bash
docker compose stop
```

```bash
docker compose down
```

```bash
docker compose down -v
```

```bash
docker compose pull
```

--------------------------------

### Silent Installation via PowerShell

Source: https://fastflowlm.com/docs/install_win

Run the installer in silent or very silent mode using PowerShell.

```powershell
.\flm-setup.exe /Silent
# or
.\flm-setup.exe /VERYSilent
```

--------------------------------

### Install Dependencies on Arch Linux

Source: https://fastflowlm.com/docs/install_lin

Installs required kernel headers, firmware, and XRT packages on Arch Linux.

```bash
sudo pacman -Syu
sudo pacman -S linux-headers linux-firmware-other xrt xrt-plugin-amdxdna

```

--------------------------------

### Example Document Content

Source: https://fastflowlm.com/docs/instructions/server/rag_LangChain

Sample text files used for the RAG knowledge base.

```text
The United Nations (UN) was officially established on 24 October 1945 after the ratification of its Charter. It replaced the League of Nations.
```

```text
UN Peacekeeping began in 1948. Since then, more than 70 operations have been deployed to help maintain peace and security.
```

```text
The Charter of the United Nations was signed on 26 June 1945 in San Francisco, at the conclusion of the United Nations Conference on International Organization.
```

--------------------------------

### Run Server Mode with ASR

Source: https://fastflowlm.com/docs/models/whisper

Start the FastFlowLM server with ASR enabled, either concurrently with an LLM or as a standalone service.

```bash
flm serve gemma3:4b --asr 1 
```

```bash
flm serve gemma3:4b -a 1 
```

```bash
flm serve --asr 1 
```

```bash
flm serve -a 1 
```

--------------------------------

### Memory Error Example

Source: https://fastflowlm.com/docs/instructions/cli

Example of an error message encountered when system DRAM is insufficient for the requested context length.

```text
[XRT] ERROR: Failed to submit the command to the hw queue (0xc01e0200):
Even after the video memory manager split the DMA buffer, the video memory manager
could not page-in all of the required allocations into video memory at the same time.
The device is unable to continue.
```

--------------------------------

### Validate NPU setup

Source: https://fastflowlm.com/docs/install_lin

Executes the validation command to check the NPU configuration.

```bash
flm validate
```

--------------------------------

### NPU validation output

Source: https://fastflowlm.com/docs/install_lin

Example output showing kernel, NPU device path, firmware version, and memory lock limits.

```text
[Linux]  Kernel: 7.0.0-rc1-00052-g27936bfca73d
[Linux]  NPU: /dev/accel/accel0
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  Memlock Limit: infinity
```

--------------------------------

### Activate FastFlowLM Server Mode

Source: https://fastflowlm.com/docs/instructions/server/msft_AI_toolkit

Command to start the local inference server for a specific model.

```powershell
flm serve llama3.2:1b
```

--------------------------------

### Start FastFlowLM Server with Embedding Model

Source: https://fastflowlm.com/docs/models/embeddinggemma

Command to launch the server with both an LLM and the embedding model enabled for concurrent use.

```powershell
flm serve gemma3:4b --embed 1 # Load the embedding model (embed-gemma:300m) in the background, with concurrent LLM loading (gemma3:4b).
```

--------------------------------

### Run CLI Mode with ASR

Source: https://fastflowlm.com/docs/models/whisper

Start the FastFlowLM CLI with an LLM and the Whisper ASR model enabled.

```bash
flm run gemma3:4b --asr 1 
```

```bash
flm run gemma3:4b -a 1 
```

--------------------------------

### Pull and List FastFlowLM Models

Source: https://fastflowlm.com/docs/instructions/server/msft_AI_toolkit

Commands to download a base model and verify its installation in the local environment.

```powershell
flm pull llama3.2:1b
```

```powershell
flm list
```

--------------------------------

### List Models in Quiet Mode

Source: https://fastflowlm.com/docs/instructions/cli

Display model lists with minimal output, optionally filtering by installation status.

```bash
flm list
flm list --quiet
flm list --filter all --quiet
flm list --filter installed --quiet
flm list --filter not-installed --quiet
```

--------------------------------

### Generate Embeddings with OpenAI Client

Source: https://fastflowlm.com/docs/models/embeddinggemma

Example of using the OpenAI Python client to interact with the FastFlowLM embedding endpoint.

```python
from openai import OpenAI

client = OpenAI(
   base_url="http://127.0.0.1:52625/v1", # FastFlowLM's local API endpoint
   api_key="flm", # Dummy key (FastFlowLM doesn't require authentication)
)

resp = client.embeddings.create(
   model="embed-gemma",
   input="Hi, everyone!"
)

print(resp.data[0].embedding)
```

--------------------------------

### Create Project Directory

Source: https://fastflowlm.com/docs/instructions/server/websearch_LangChain

Commands to initialize the project folder.

```bash
mkdir rag_websearch_flm
cd rag_websearch_flm
```

--------------------------------

### Run Qwen2.5-VL-3B-Instruct

Source: https://fastflowlm.com/docs/models/qwen

Launch the Qwen2.5-VL-3B-Instruct vision model using the FastFlowLM CLI.

```powershell
flm run qwen2.5vl-it:3b
```

--------------------------------

### Run Qwen2.5-3B-Instruct

Source: https://fastflowlm.com/docs/models/qwen

Launch the Qwen2.5-3B-Instruct model using the FastFlowLM CLI.

```powershell
flm run qwen2.5-it:3b
```

--------------------------------

### Launch Notepad for Configuration

Source: https://fastflowlm.com/docs/instructions/server/webui

Command to open the docker-compose file in Notepad.

```bash
notepad docker-compose.yaml
```

--------------------------------

### Show CLI Help

Source: https://fastflowlm.com/docs/instructions/cli

Displays available commands and usage information for the CLI.

```bash
flm help
```

--------------------------------

### Create Virtual Environment

Source: https://fastflowlm.com/docs/instructions/server/rag_LangChain

Commands to initialize a Python virtual environment on Windows.

```bash
mkdir rag_with_flm
cd rag_with_flm
python -m venv rag-env
.\rag-env\Scripts\activate
```

--------------------------------

### Run Qwen3-VL-4B-Instruct in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the Qwen3-VL-4B-Instruct model using the FastFlowLM CLI.

```powershell
flm run qwen3vl-it:4b
```

--------------------------------

### Set Host at Launch

Source: https://fastflowlm.com/docs/instructions/cli

Specify a custom host address for the server session.

```bash
flm serve llama3.2:1b --host 127.0.0.1
```

--------------------------------

### Rebuild DKMS Module on Arch Linux

Source: https://fastflowlm.com/docs/install_lin

Rebuilds and installs the DKMS module for the current kernel.

```bash
sudo dkms autoinstall -k "$(uname -r)"
sudo depmod -a
sudo reboot

```

--------------------------------

### Run Qwen3.5-0.8B in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the Qwen3.5-0.8B model using the FastFlowLM CLI.

```bash
flm run qwen3.5:0.8b
```

--------------------------------

### Set Context Length

Source: https://fastflowlm.com/docs/instructions/cli

Configure the context length at launch using --ctx-len or -c.

```bash
flm run llama3.2:1b --ctx-len 8192
flm serve llama3.2:1b --ctx-len 8192
```

--------------------------------

### Run Qwen3-4B-Instruct-2507 in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the Qwen3-4B-Instruct-2507 model using the FastFlowLM CLI.

```powershell
flm run qwen3-it:4b
```

--------------------------------

### Run Qwen3.5-2B in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the Qwen3.5-2B model using the FastFlowLM CLI.

```bash
flm run qwen3.5:2b
```

--------------------------------

### Run Qwen3.6-35B-A3B in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the model using the FastFlowLM command-line interface.

```powershell
flm run qwen3.6-moe:35b-a3b
```

--------------------------------

### Configure Image Resizing for Qwen3.5-0.8B

Source: https://fastflowlm.com/docs/models/qwen

Control image resolution during execution or serving using the -r flag.

```bash
flm run qwen3.5:0.8b -r 1
```

```bash
flm serve qwen3.5:0.8b -r 1
```

--------------------------------

### Run Llama 3.1 8B Instruct

Source: https://fastflowlm.com/docs/models/llama

Execute the Llama 3.1 8B Instruct model using the FastFlowLM CLI.

```powershell
flm run llama3.1:8b
```

--------------------------------

### Configure Image Resizing for Qwen3.5-2B

Source: https://fastflowlm.com/docs/models/qwen

Control image resolution during execution or serving using the -r flag.

```bash
flm run qwen3.5:2b -r 1
```

```bash
flm serve qwen3.5:2b -r 1
```

--------------------------------

### List Available Models

Source: https://fastflowlm.com/docs/instructions/cli

Lists all locally downloaded and available models.

```text
/list
```

--------------------------------

### Run Llama 3.2 3B Instruct

Source: https://fastflowlm.com/docs/models/llama

Execute the Llama 3.2 3B Instruct model using the FastFlowLM CLI.

```powershell
flm run llama3.2:3b
```

--------------------------------

### Configure Image Resizing for Qwen3.5-4B

Source: https://fastflowlm.com/docs/models/qwen

Set image height during execution or serving using the -r flag.

```powershell
flm run qwen3.5:4b -r 1
```

```powershell
flm serve qwen3.5:4b -r 1
```

--------------------------------

### Execute RAG Script

Source: https://fastflowlm.com/docs/instructions/server/rag_LangChain

Command to run the demonstration script.

```bash
python rag_demo.py
```

--------------------------------

### Set Server Port at Launch

Source: https://fastflowlm.com/docs/instructions/cli

Specify a custom port for the current server session.

```bash
flm serve llama3.2:1b --port 8000
flm serve llama3.2:1b -p 8000
```

--------------------------------

### Run GPT-OSS Models via CLI

Source: https://fastflowlm.com/docs/models/gpt-oss

Commands to launch the gpt-oss-20b and gpt-oss-safeguard-20b models using the FastFlowLM CLI.

```bash
flm run gpt-oss:20b
```

```bash
flm run gpt-oss-safeguard:20b
```

--------------------------------

### Configure Image Resizing for Qwen3.5-9B

Source: https://fastflowlm.com/docs/models/qwen

Set image height during execution or serving using the -r flag.

```powershell
flm run qwen3.5:9b -r 1
```

```powershell
flm serve qwen3.5:9b -r 1
```

--------------------------------

### Run Qwen3.5-9B in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the Qwen3.5-9B model using the FastFlowLM CLI.

```powershell
flm run qwen3.5:9b
```

--------------------------------

### List Models

Source: https://fastflowlm.com/docs/instructions/cli

Displays all available and downloaded models. Supports JSON output format.

```bash
flm list
```

```bash
flm list --json
```

--------------------------------

### Run Qwen3.5-4B in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the Qwen3.5-4B model using the FastFlowLM CLI.

```powershell
flm run qwen3.5:4b
```

--------------------------------

### Run Qwen3 Models via PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Commands to launch specific Qwen3 model variants using the FastFlowLM CLI.

```powershell
flm run qwen3:0.6b
```

```powershell
flm run qwen3:1.7b
```

```powershell
flm run qwen3:4b
```

--------------------------------

### Summarization with FastFlowLM and LangChain

Source: https://fastflowlm.com/docs/instructions/server/websearch_LangChain

Summarizes search results using an OllamaLLM instance and a PromptTemplate.

```python
def summarize_with_fastflowlm(search_results: str, model_name=MODEL_NAME) -> str:
    if not search_results.strip():
        return "⚠️ No search results to summarize."

    llm = OllamaLLM(model=model_name, base_url=BASE_URL)

    prompt = PromptTemplate.from_template("""
You are a factual and concise research assistant. Summarize the following web search results clearly and accurately.

Search Results:
{search_results}

Summary:
""")

    try:
        summary = (prompt | llm).invoke({"search_results": search_results})
    except Exception as e:
        return f"❌ Error generating summary: {e}"

    return summary.strip()
```

--------------------------------

### Check a Downloaded Model

Source: https://fastflowlm.com/docs/instructions/cli

Verify the file hashes for a downloaded model.

```bash
flm check llama3.2:3b
```

--------------------------------

### Main Execution Flow

Source: https://fastflowlm.com/docs/instructions/server/websearch_LangChain

Orchestrates the web search, summarization, vector store creation, and RAG query process.

```python
def main():
    search_query = "Recent developments in AMD Ryzen AI chips"
    follow_up_question = "What new features do AMD Ryzen AI chips have in 2025?"

    # Web Search
    search_text = run_web_search(search_query, max_results=MAX_RESULTS)
    print("\n🌐 Raw Web Search Output:\n")
    print(search_text)

    # Summarization
    summary = summarize_with_fastflowlm(search_text)
    print("\n🧠 Summary:\n")
    print(summary)

    # Build RAG Vector DB
    retriever = build_retriever_from_text(summary)

    # RAG Follow-up
    ask_rag_question(follow_up_question, retriever)


if __name__ == "__main__":
    main()
```

--------------------------------

### Enable ASR

Source: https://fastflowlm.com/docs/instructions/cli

Load an ASR model concurrently with an LLM.

```bash
flm run gemma3:4b --asr 1
flm serve gemma3:4b --asr 1
```

--------------------------------

### Input Audio in CLI

Source: https://fastflowlm.com/docs/models/whisper

Provide an audio file path to the CLI for processing.

```bash
/input "path\to\audio_sample.mp3" summarize it
```

--------------------------------

### Batch Processing Prompts with OpenAI Client

Source: https://fastflowlm.com/docs/instructions/server/openapi

Iterate through a list of prompts to generate responses sequentially. Requires the openai library and a running FastFlowLM server instance.

```python
# Batched prompts
from openai import OpenAI

prompts = [
    "Summarize the causes of World War I.",
    "Describe how a transistor works.",
    "What are the key themes in ‘To Kill a Mockingbird’?",
]

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="flm")

for prompt in prompts:
    response = client.chat.completions.create(
        model="llama3.2:1b",
        messages=[
            {"role": "system", "content": "You are a concise academic tutor."},
            {"role": "user", "content": prompt}
        ]
    )
    print(f"\n📝 Prompt: {prompt}\n🔍 Answer: {response.choices[0].message.content}")

# cleanup
del response, client
import gc
gc.collect()
```

--------------------------------

### Run Phi-4-mini-instruct in PowerShell

Source: https://fastflowlm.com/docs/models/phi

Execute the model using the FastFlowLM CLI in a PowerShell environment.

```powershell
flm run phi4-mini-it:4b
```

--------------------------------

### Configure Image Resizing for Qwen2.5-VL

Source: https://fastflowlm.com/docs/models/qwen

Set image resizing options when running or serving the vision model using the -r flag.

```powershell
flm run qwen2.5vl-it:3b -r 1
```

```powershell
flm serve qwen2.5vl-it:3b -r 1
```

--------------------------------

### Stream Model Output with OpenAI Client

Source: https://fastflowlm.com/docs/instructions/server/openapi

Demonstrates how to process token-by-token streaming responses from the model. Requires the openai library and manual garbage collection for cleanup.

```python
# Streaming
from openai import OpenAI
import gc, sys

client = OpenAI(base_url="http://127.0.0.1:52625/v1/", api_key="flm")

stream = client.chat.completions.create(
    model="llama3.2:1b",
    messages=[
        {"role": "system", "content": "You are a fast, concise assistant."},
        {"role": "user", "content": "Summarize the plot of Hamlet in 3 sentences."}
    ],
    stream=True
)

print("=== STREAMING BEGIN ===")
try:
    for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            print(content, end="", flush=True)
    print("\n=== STREAMING END ===")
except Exception as e:
    sys.stderr.write(f"\n[STREAM ERROR] {e}\n")

#cleanup
del stream, client
gc.collect()
```

--------------------------------

### Run Benchmarking Tool

Source: https://fastflowlm.com/docs/instructions/cli

Executes performance benchmarks for a specific model and displays the results.

```text
flm bench llama3.2:1b
```

```text
[FLM]  === Benchmark Results ===

 Context Length |              TTFT (s) |      Prefill Speed (tok/s) |     Decoding Speed (tok/s)
----------------------------------------------------------------------------------------------------
             1k |       0.815 ±   0.012 |        1233.57 ±     17.91 |       61.39 ±      0.46
             2k |       1.144 ±   0.035 |        1728.16 ±     55.68 |       59.04 ±      0.36
             4k |       1.955 ±   0.009 |        2001.55 ±      9.22 |       54.32 ±      0.47
             8k |       3.821 ±   0.010 |        2037.62 ±      5.40 |       46.89 ±      0.19
            16k |       9.144 ±   0.011 |        1698.20 ±      2.08 |       37.20 ±      0.14
            32k |      26.346 ±   0.037 |        1177.16 ±      1.53 |       26.38 ±      0.07
----------------------------------------------------------------------------------------------------
```

--------------------------------

### Run Qwen3-8B in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the Qwen3-8B model using the FastFlowLM CLI.

```powershell
flm run qwen3:8b
```

--------------------------------

### Run the Script

Source: https://fastflowlm.com/docs/instructions/server/websearch_LangChain

Command to execute the Python script.

```bash
python websearch_rag.py
```

--------------------------------

### Run with Image Resizing

Source: https://fastflowlm.com/docs/models/qwen

Apply image resizing during model execution using the -r flag to optimize performance based on resolution requirements.

```powershell
flm run qwen3.6-moe:35b-a3b -r 1
```

```powershell
flm run qwen3.6-moe:35b-a3b -r 1
```

--------------------------------

### Model List File Path

Source: https://fastflowlm.com/docs/instructions/cli

Location of the model list configuration file.

```text
C:\Program Files\flm\model_list.json
```

--------------------------------

### Send Multi-Modal Input to Gemma 4

Source: https://fastflowlm.com/docs/instructions/server/openapi

Uses the OpenAI client to send a text prompt alongside Base64-encoded audio and image data. Requires the local server to be running at the specified base URL.

```python
import base64
import gc
import sys
from openai import OpenAI

audio_path = r"C:\Users\info\OneDrive\Desktop\FLM\audio_test\audio.wav"
image_path = r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image.png"


# Read the audio and image files and encode them as Base64 for API input
with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")
with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")


client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

stream = client.chat.completions.create(
    model="gemma4-it:e2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Finish two tasks: 1. Summarize the audio. 2. Describe the image."},
                {
                    "type": "input_audio",  
                    "input_audio": {"data": audio},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        }
    ],
    stream=True
)

print("=== STREAMING BEGIN ===")
try:
    for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            print(content, end="", flush=True)
    print("\n=== STREAMING END ===")
except Exception as e:
    sys.stderr.write(f"\n[STREAM ERROR] {e}\n")

# cleanup
del stream, client
gc.collect()
```

--------------------------------

### Prompting with External Text Files

Source: https://fastflowlm.com/docs/instructions/server/openapi

Shows how to read a local text file and use its content as a prompt for the model. Suitable for large context windows supported by FastFlowLM.

```python
# Use a text file to prompt
from openai import OpenAI

with open("C:\\Users\\<username>\\Downloads\\alice_in_wonderland.txt", "r", encoding="utf-8") as f:
    user_prompt = f.read()

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="flm")

response = client.chat.completions.create(
    model="llama3.2:1b",
    messages=[
        {"role": "system", "content": "You are a story rewriter."},
        {"role": "user", "content": user_prompt}
    ]
)

print(response.choices[0].message.content)

# cleanup
del response, client
import gc
gc.collect()
```

--------------------------------

### Docker Compose Configuration

Source: https://fastflowlm.com/docs/instructions/server/webui

Configuration file for running Open WebUI with FastFlowLM integration.

```yaml
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui-data:/app/backend/data
    environment:
      # Point WebUI to FLM's OpenAI-compatible server
      - OPENAI_API_BASE_URL=http://host.docker.internal:52625/v1
      - OPENAI_API_KEY=dummy-key

      # Explicitly disable Ollama auto-connect
      - OLLAMA_BASE_URL=

      # WebUI settings
      - WEBUI_AUTH=false
      - WEBUI_SECRET_KEY=dummysecretkey
      - ENABLE_TITLE_GENERATION=false
      - ENABLE_FOLLOW_UP_GENERATION=false
      - ENABLE_TAGS_GENERATION=false
      - ENABLE_RETRIEVAL_QUERY_GENERATION=false
      - ENABLE_IMAGE_PROMPT_GENERATION=false
      - ENABLE_WEB_SEARCH=false
      - ENABLE_SEARCH_QUERY_GENERATION=false
    restart: unless-stopped

volumes:
  open-webui-data:
```

--------------------------------

### Show Server Port

Source: https://fastflowlm.com/docs/instructions/server

Display the currently configured default server port.

```powershell
  flm port
```

--------------------------------

### Set Configuration Variables

Source: https://fastflowlm.com/docs/instructions/cli

Configures decoding parameters or sets specific limits like generation length.

```text
/set
```

```text
/set gen-lim 128
```

--------------------------------

### Run Qwen3-4B-Thinking-2507 in PowerShell

Source: https://fastflowlm.com/docs/models/qwen

Execute the Qwen3-4B-Thinking-2507 model using the FastFlowLM CLI.

```powershell
flm run qwen3-tk:4b
```

--------------------------------

### Run LiquidAI LFM Models via PowerShell

Source: https://fastflowlm.com/docs/models/lfm

Use these commands to launch specific LFM model variants in the FastFlowLM environment.

```powershell
flm run lfm2:1.2b
```

```powershell
flm run lfm2:2.6b
```

```powershell
flm run lfm2-trans:2.6b
```

```powershell
flm run lfm2.5-it:1.2b
```

```powershell
flm run lfm2.5-tk:1.2b
```

--------------------------------

### Configure Image Resize Options

Source: https://fastflowlm.com/docs/models/qwen

Set image height for vision models using the -r flag during run or serve commands.

```powershell
flm run qwen3vl-it:3b -r 1
```

```powershell
flm serve qwen3vl-it:3b -r 1
```

--------------------------------

### Run Gemma Models via PowerShell

Source: https://fastflowlm.com/docs/models/gemma

Commands to launch specific Gemma model versions using the FastFlowLM CLI.

```powershell
flm run gemma3:1b
```

```powershell
flm run gemma3:4b
```

```powershell
flm run gemma4-it:e2b
```

```powershell
flm run gemma4-it:e4b
```

--------------------------------

### Project Directory Structure

Source: https://fastflowlm.com/docs/instructions/server/rag_LangChain

Recommended file layout for the RAG project.

```text
rag_with_flm/
├── docs/
│   ├── un_history.txt
│   ├── peacekeeping.txt
│   └── founding_charter.txt
├── rag_demo.py
├── rag-env/
```

--------------------------------

### Attach Multimodal Inputs in CLI Mode

Source: https://fastflowlm.com/docs/models/gemma

Commands to provide image or audio files as input to the model during a CLI session.

```text
/input "file/to/image.jpg" describe this image.
```

```text
/input "file/to/audio.mp3" summarize this audio.
```

--------------------------------

### Build Static FastFlowLM Binary

Source: https://fastflowlm.com/docs/install_lin

Builds a self-contained binary that bundles XRT and XDNA dependencies.

```bash
cd src
cmake --preset linux-static
cmake --build build -j$(nproc)
sudo cmake --install build

```

--------------------------------

### Configure Model Storage Path on Linux

Source: https://fastflowlm.com/docs/instructions/cli

Sets the environment variable to change the default model storage location. Includes commands for temporary and permanent configuration.

```bash
export FLM_MODEL_PATH="/your/custom/path"
```

```bash
echo 'export FLM_MODEL_PATH="/your/custom/path"' >> ~/.bashrc
source ~/.bashrc
```

--------------------------------

### Validate System Compatibility

Source: https://fastflowlm.com/docs/instructions/cli

Checks hardware requirements for FastFlowLM. Use the --json flag to output results in JSON format.

```bash
flm validate
```

```bash
flm validate --json
```

--------------------------------

### Implement get_horoscope tool calling

Source: https://fastflowlm.com/docs/instructions/server/tool_calling

A complete workflow for defining a tool, requesting a tool call from the model, executing the function locally, and providing the result back to the model.

```python
from openai import OpenAI
import json

# Connect to local FastFlowLM server
client = OpenAI(
    base_url="http://127.0.0.1:1234/v1",
    api_key="flm",
)

# OpenAI Chat Completions Tool definition
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_horoscope",
            "description": "Get today's horoscope for an astrological sign.",
            "parameters": {
                "type": "object",
                "properties": {
                    "sign": {
                        "type": "string",
                        "description": "An astrological sign like Taurus or Aquarius",
                    },
                },
                "required": ["sign"],
            },
        },
    },
]

# Tool implementation
def get_horoscope(sign: str) -> str:
    return f"{sign}: Next Tuesday you will befriend a baby otter."

# Step 1:
## Original user message
messages = [
    {"role": "user", "content": "What is my horoscope? I am an Aquarius."}
]

## Ask model; it should emit a tool call
resp1 = client.chat.completions.create(
    model="qwen3-it:4b",
    messages=messages,
    tools=tools,
)
assistant_msg = resp1.choices[0].message

# Step 2:
## Add assistant message (with tool calls) to messages
messages.append(
    {
        "role": "assistant",
        "content": assistant_msg.content,      # may be None
        "tool_calls": assistant_msg.tool_calls # tool_calls
    }
)

## Execute tool calls and add tool call outputs back
if assistant_msg.tool_calls:
    for tc in assistant_msg.tool_calls:
        if tc.function.name == "get_horoscope":
            args = json.loads(tc.function.arguments)  # {"sign": "..."}
            horoscope = get_horoscope(args["sign"])
            messages.append(
                {
                    "role": "tool",
                    "tool_call_id": tc.id,
                    "content": json.dumps({"horoscope": horoscope}),
                }
            )

print("Messages so far:")
print(messages)

# Step 3:
## Combine all the messages and ask again
resp2 = client.chat.completions.create(
    model="qwen3-it:4b",
    messages=messages,
    tools=tools,
)

print("\nFinal output:")
print(resp2.choices[0].message.content)
```

--------------------------------

### Clone FastFlowLM Repository

Source: https://fastflowlm.com/docs/install_lin

Downloads the source code from GitHub.

```bash
git clone https://github.com/FastFlowLM/FastFlowLM.git
cd FastFlowLM

```

--------------------------------

### Add AMD XRT PPA on Ubuntu

Source: https://fastflowlm.com/docs/install_lin

Adds the required PPA for XRT and NPU support on Ubuntu 24.04 and 25.10.

```bash
sudo add-apt-repository ppa:lemonade-team/stable
sudo apt update

```

--------------------------------

### Run DeepSeek-R1-0528-Qwen3-8B in PowerShell

Source: https://fastflowlm.com/docs/models/deepseek

Executes the DeepSeek-R1-0528-Qwen3-8B model using the FastFlowLM CLI.

```powershell
flm run deepseek-r1-0528:8b
```

--------------------------------

### Load Image File

Source: https://fastflowlm.com/docs/instructions/cli

Loads an image file for vision models. Supported formats are .png and .jpg.

```text
/input "<image_path>" prompt
```

```text
/input "C:\Users\Public\Desktop\cat.jpg" describe this image
```

--------------------------------

### Run DeepSeek-R1-Distill-Llama-8B in PowerShell

Source: https://fastflowlm.com/docs/models/deepseek

Executes the DeepSeek-R1-Distill-Llama-8B model using the FastFlowLM CLI.

```powershell
flm run deepseek-r1:8b
```

--------------------------------

### Set Context Length

Source: https://fastflowlm.com/docs/instructions/server

Configure the context length at launch. Values are rounded up to the nearest power of 2 with a minimum of 512.

```powershell
flm serve llama3.2:1b --ctx-len 8192
```

--------------------------------

### Transcribe Audio via OpenAI Python SDK

Source: https://fastflowlm.com/docs/models/whisper

Use the official OpenAI Python client to send audio files to the local FastFlowLM server for transcription.

```python
# Import the official OpenAI Python SDK (FastFlowLM mirrors the OpenAI API schema)
from openai import OpenAI

# Initialize the client to point at your local FastFlowLM server
# - base_url: FastFlowLM's local OpenAI-compatible REST endpoint
# - api_key: Dummy token; FastFlowLM typically doesn't enforce auth, but the client requires a string
client = OpenAI(
    base_url="http://127.0.0.1:52625/v1",  # FastFlowLM local API endpoint
    api_key="flm",                         # Placeholder key
)

# Open the audio file in binary mode and create a transcription request
# - model: name of the speech-to-text model exposed by FLM (e.g., "whisper-v3")
# - file: file-like object pointing to your audio
with open("audio.mp3", "rb") as f:
    resp = client.audio.transcriptions.create(
        model="whisper-v3",
        file=f,
    )

# Print the transcribed text returned by the server
print(resp.text)
```

--------------------------------

### Display Runtime Statistics

Source: https://fastflowlm.com/docs/instructions/cli

Displays metrics such as token count and throughput.

```text
/status
```
Tokens:
Raw