GLM-V

https://github.com/zai-org/glm-v
Admin
GLM-V is an open-source repository containing GLM-4.5V and GLM-4.1V series vision-language models,...
Tokens:43,038
Snippets:432
Trust Score:7.6
License:Apache-2.0
Update:3 months ago
Show doc for...
Tokens:
Raw
### Quick Start Commands

Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README.md

Standard commands to install dependencies and launch the development environment.

```bash
pnpm install
```

```bash
pnpm dev
```

--------------------------------

### Install the package

Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md

Install the reward system package in editable mode.

```bash
pip install -e .
```

--------------------------------

### Install Dependencies and Run Demo

Source: https://github.com/zai-org/glm-v/blob/main/examples/midscene-ts-demo/README.md

Commands to install project dependencies, download the required browser, and execute the demo script.

```bash
npm install
# Puppeteer needs download a compatible Chrome
npx puppeteer browsers install chrome
# run demo.ts
npx tsx demo.ts
```

--------------------------------

### Quick Start Command

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prd-to-app/SKILL.md

Command to initialize the project from a clean state.

```bash
bash start.sh
```

--------------------------------

### Install Environment Dependencies

Source: https://github.com/zai-org/glm-v/blob/main/README.md

Install required packages from the requirements file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Run Test Suite

Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md

Execute the test suite to verify the installation.

```bash
pytest tests/
```

--------------------------------

### Authenticate and Install Dependencies

Source: https://github.com/zai-org/glm-v/blob/main/examples/AMD_GPU/README.md

Logs into Hugging Face and installs the required flash-attn and model-specific transformer libraries.

```shell
huggingface-cli login
```

```shell
pip install flash-attn
pip install transformers-v4.55.0-GLM-4.5V-preview
```

--------------------------------

### Message Organization Example

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Example structure for organizing messages when historical steps are present.

```python
[{'role': 'user',
  'content': [{'type': 'text',
    'text': 'You are a GUI Agent, and your primary task is to respond accurately to user requests or questions. In addition to directly answering the user\'s queries, you can also use tools or perform GUI operations directly until you fulfill the user\'s request or provide a correct answer. You should carefully read and understand the images and questions provided by the user, and engage in thinking and reflection when appropriate.
```

--------------------------------

### Example Agent Action Execution

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent.md

A sample action command to open the Broccoli application.

```python
{'action_type': 'open_app', 'app_name': 'Broccoli'}
```

--------------------------------

### Example Output Format

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prompt-gen/SKILL.md

The expected structure of the generated prompt output.

```text
### Content Analysis
A cyberpunk cityscape at night, with dense skyscrapers, glowing neon signs, and rain-wet streets reflecting colorful light.

### Prompt
Cyberpunk cityscape at night, towering skyscrapers with glowing neon signs,
rain-wet streets reflecting colorful lights, flying cars in the distance,
volumetric fog, dramatic lighting, ultra detailed, 8K, cinematic composition

### Prompt Breakdown
- **Subject**: Futuristic skyline with skyscrapers and neon lights
- **Style**: Cyberpunk, sci-fi
- **Color**: Cool/warm contrast with blue-purple dominance and neon accents
- **Lighting**: Neon glow, wet-surface reflections, volumetric fog
- **Composition**: Wide-angle perspective with layered depth
- **Mood**: Mysterious, futuristic, high-tech
```

--------------------------------

### Install Runtime Dependencies

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-grounding/SKILL.md

Install the necessary Python packages required for the GLM-V skill functionality.

```bash
pip install -r scripts/requirements.txt
```

--------------------------------

### Reward System Configuration

Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md

Example YAML configuration for the reward system, including verifier settings.

```yaml
reward_configs:
  math_verifier_config:
    verifier_type: "math"
    enable_llm_judge_fallback: true
    llm_judge_url:
      - "https://open.bigmodel.cn/api/paas/v4/chat/completions"
```

--------------------------------

### View Markdown Output Example

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-doc-based-writing/SKILL.md

Represents the expected structure of the generated Markdown content.

```markdown
XXX
```

--------------------------------

### Deployment Script Template

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prd-to-app/SKILL.md

A self-contained bash script template for initializing dependencies, databases, and starting the application services.

```bash
#!/bin/bash
set -e

echo "=== PRD-to-App: Starting deployment ==="

# 1. System dependencies
echo "[1/7] Checking system dependencies..."
# Install Node.js, npm, etc. if missing
# Install database if needed

# 2. Backend setup
echo "[2/7] Setting up backend..."
cd /workspace/backend
npm install  # or pip install -r requirements.txt

# 3. Database initialization
echo "[3/7] Initializing database..."
# Drop existing DB / reset
# Run migrations
# Load seed data

# 4. Frontend setup
echo "[4/7] Setting up frontend..."
cd /workspace/frontend
npm install

# 5. Copy resources
echo "[5/7] Copying resource files..."
# cp -r /workspace/resources/* /workspace/frontend/public/assets/

# 6. Start backend
echo "[6/7] Starting backend server..."
cd /workspace/backend
npm run dev &   # or equivalent
BACKEND_PID=$!

# 7. Start frontend
echo "[7/7] Starting frontend..."
cd /workspace/frontend
PORT=3000 npm run dev &
FRONTEND_PID=$!

echo ""
echo "=== Application is running at http://localhost:3000 ==="
echo "Backend PID: $BACKEND_PID"
echo "Frontend PID: $FRONTEND_PID"
echo "To stop: kill $BACKEND_PID $FRONTEND_PID"

wait
```

--------------------------------

### Open App Action Definition

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Defines the open_app action to launch installed applications.

```text
Calling rule: `{"action_type": "open_app", "app_name": "<name>"}`
```

```json
{
    "name": "open_app",
    "description": "Open an app (nothing will happen if the app is not installed).",
    "parameters": {
        "type": "object",
        "properties": {
            "app_name": {
                "type": "string",
                "description": "The name of the app to open. Supported apps: <supported_apps>"
            }
        },
        "required": [
            "app_name"
        ]
    }
}
```

--------------------------------

### Install Midscene CLI

Source: https://github.com/zai-org/glm-v/blob/main/examples/midscene-yaml-demo/README.md

Install the required Midscene CLI package via npm.

```shell
npm i @midscene/cli
```

--------------------------------

### Map Prototype Card to Database Entity

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prd-to-app/references/seed_data_guide.md

Example mapping of a product card from a prototype to a database table structure.

```text
Prototype Card "iPhone 15 Pro - ¥8999 - 128GB"
  → products table:
    name: "iPhone 15 Pro"
    price: 8999.00
    spec: "128GB"
    image: "resources/iphone15pro.jpg"  (if exists)
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-pdf-to-web/SKILL.md

Required Python packages for processing PDFs and images.

```bash
pip install pymupdf pillow
```

--------------------------------

### README Template Structure

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prd-to-app/SKILL.md

Standard template for the project README file including overview, technology stack, and quick start instructions.

```markdown
# [Project Name]

## Overview
[From PRD product overview]

## Technology Stack
- Frontend: ...
- Backend: ...
- Database: ...

## Quick Start
```bash
bash start.sh
```
Then visit http://localhost:3000

## Project Structure
[Directory tree]

## Database
[Schema overview, how to re-seed]

## API Reference
[Key endpoints]
```

--------------------------------

### Initialize the project environment

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-web-replication/SKILL.md

Sets up the blueprint directory and configures the browser session for the target website.

```bash
# Create the blueprint root directory
mkdir -p blueprint/_assets
# Open the browser and visit the target site
agent-browser open <target URL>
agent-browser set viewport 1920 1080
agent-browser wait --load networkidle
```

--------------------------------

### Run the demo

Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md

Execute the reward system demonstration script.

```bash
python examples/reward_system_demo.py
```

--------------------------------

### Initialize and Use Reward System

Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md

Initialize the RewardSystem with a configuration file and evaluate model responses.

```python
from glmv_reward import RewardSystem

# Initialize the reward system
reward_system = RewardSystem("examples/configs/example.yaml")

# Evaluate model responses
rewards = reward_system.get_reward(
    prompts=["What is 15 + 27?"],
    answers=["<think>15 + 27 = 42</think><answer><|begin_of_box|>42<|end_of_box|></answer>"],
    gt_answers=["<think>15 + 27 = 42</think><answer><|begin_of_box|>42<|end_of_box|></answer>"],
    datasources=["math"]
)

# Use reward in your RL training
print(f"Reward: {rewards[0]}")  # Output: 1.0 (correct answer)
```

--------------------------------

### Configure the reward system

Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md

Copy the template configuration file to create a local configuration.

```bash
cp examples/configs/example.yaml.template examples/configs/example.yaml
```

--------------------------------

### Define Scroll Action Rule

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Schema for the scroll action, which requires start coordinates and direction.

```json
{
    'name': 'scroll',
    'description': 'Scroll an element at the specified coordinates in the specified direction by a given number of wheel steps.',
    'parameters': {
        'type': 'object',
        'properties': {
            'start_box': {
                'type': 'array',
                'items': {
                    'type': 'integer'
                },
                'description': 'Coordinates [x,y] of the element or area to scroll, normalized to 0-999 range.'
            },
            'direction': {
                'type': 'string',
                'enum': ['down', 'up'],
                'description': 'The direction to scroll: 'down' or 'up'.'
            },
            'step': {
                'type': 'integer',
                'default': 5,
                'description': 'Number of wheel steps to scroll, default is 5.'
            },
            'element_info': {
                'type': 'string',
                'description': 'Optional text description of the UI element being scrolled.'
            }
        },
        'required': ['start_box', 'direction']
    }
}
```

--------------------------------

### Standard Project Directory Structure

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prd-to-app/SKILL.md

Recommended file and folder organization for a full-stack application project.

```text
/workspace/
├── frontend/          ← or client/
│   ├── src/
│   │   ├── components/
│   │   ├── pages/
│   │   ├── hooks/
│   │   ├── services/   ← API client
│   │   ├── types/
│   │   └── assets/     ← copied from /workspace/resources
│   └── ...
├── backend/           ← or server/
│   ├── src/
│   │   ├── routes/
│   │   ├── models/
│   │   ├── controllers/
│   │   ├── middleware/
│   │   └── seed/
│   └── ...
├── docs/
│   ├── design.md
│   └── README.md
├── start.sh
└── prd.md
```

--------------------------------

### PDF Page Mapping Output

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-pdf-to-web/SKILL.md

Example of the JSON output format produced by the PDF conversion script.

```json
[{"page": 1, "path": "/abs/path/page_001.png"}, ...]
```

--------------------------------

### Mouse Drag Action

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md

Drag the mouse from a starting coordinate to an ending coordinate while holding the left button.

```json
{
    'name': 'left_drag',
    'description': 'Drag the mouse from starting coordinates to ending coordinates while holding the left mouse button.',
    'parameters': {
        'type': 'object',
        'properties': {
            'start_box': {
                'type': 'array',
                'items': {
                    'type': 'integer'
                },
                'description': 'Starting coordinates [x1,y1] for the drag operation, normalized to 0-999 range.'
            },
            'end_box': {
                'type': 'array',
                'items': {
                    'type': 'integer'
                },
                'description': 'Ending coordinates [x2,y2] for the drag operation, normalized to 0-999 range.'
            },
            'element_info': {
                'type': 'string',
                'description': 'Optional text description of the UI element being dragged.'
            }
        },
        'required': ['start_box', 'end_box']
    }
}
```

--------------------------------

### Build Commands

Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README.md

Commands to package the application for various platforms or build without packaging.

```bash
# Build all platforms
pnpm build

# Build Windows
pnpm build:win

# Build macOS
pnpm build:mac

# Build Linux
pnpm build:linux

# Build without packaging
pnpm build:unpack
```

--------------------------------

### Define Open App Action

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent.md

Use to launch a supported application by name.

```json
{"action_type": "open_app", "app_name": <name>}
```

--------------------------------

### Navigate Home Action

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Returns the agent to the device home screen.

```json
{
    "name": "navigate_home",
    "description": "Navigate to the home screen.",
    "parameters": {
        "type": "object",
        "properties": {},
        "required": []
    }
}
```

--------------------------------

### Launch Rocm-vllm Docker Container

Source: https://github.com/zai-org/glm-v/blob/main/examples/AMD_GPU/README.md

Starts a privileged Docker container with necessary device access for ROCm and vLLM.

```shell
docker run -d -it --ipc=host --network=host --privileged --cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri --device=/dev/mem --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /:/work -e SHELL=/bin/bash  --name vllm_GLM_V  rocm/vllm-dev:nightly
```

--------------------------------

### open_app

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Open a specified app.

```APIDOC
## open_app

### Description
Open an app (nothing will happen if the app is not installed).

### Request Body
- **app_name** (string) - Required - The name of the app to open.

### Request Example
{"action_type": "open_app", "app_name": "settings"}
```

--------------------------------

### Define Data Model Schema

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prd-to-app/SKILL.md

Example format for defining database table structures and mapping prototype elements to fields.

```text
products table:
  id          SERIAL PRIMARY KEY
  name        VARCHAR(200) NOT NULL    ← from product card title
  price       DECIMAL(10,2) NOT NULL   ← from product card price label
  image_url   TEXT NOT NULL             ← from product card image
  category_id INTEGER REFERENCES categories(id) ← from category filter
  ...
```

--------------------------------

### Development Workflow Commands

Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README.md

Comprehensive list of commands for maintaining, testing, and building the project.

```bash
# Install dependencies
pnpm install

# Development mode
pnpm dev

# Type checking
pnpm typecheck

# Lint
pnpm lint

# Format code
pnpm format

# Rebuild native modules
pnpm rebuild

# Build the project
pnpm build

# Build with type checking
pnpm build:with-typecheck
```

--------------------------------

### Example Memory JSON Format

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

The memory component must be stored in a strict JSON format containing only task-relevant information extracted from the webpage.

```json
{"user_email_address": "test@163.com", "user_email_password": "123456", "jack_email_address": "jack@163.com"}
```

--------------------------------

### CLI Usage Reference

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-resume-screen/SKILL.md

General syntax for the resume screening command.

```bash
python scripts/resume_screen.py --files FILE [FILE...] --criteria CRITERIA [OPTIONS]
```

--------------------------------

### Define Navigation and Keyboard Actions

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent.md

Use for basic system-level navigation and key presses.

```json
{"action_type": "keyboard_enter"}
```

```json
{"action_type": "navigate_home"}
```

```json
{"action_type": "navigate_back"}
```

--------------------------------

### CLI Usage Reference

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-caption/SKILL.md

General syntax for the captioning script.

```bash
python {baseDir}/scripts/glmv_caption.py (--images IMG [IMG...] | --videos VID [VID...] | --files FILE [FILE...]) [OPTIONS]
```

--------------------------------

### Deployment Script Execution

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prd-to-app/SKILL.md

Commands to make the generated deployment script executable and run it.

```bash
chmod +x /workspace/start.sh
```

```bash
bash /workspace/start.sh
```

--------------------------------

### Configure Thinking Budget

Source: https://github.com/zai-org/glm-v/blob/main/README.md

Import the custom logit processor and define the thinking budget parameters for SGLang.

```python
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
```

```python
extra_body={
            "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
            "custom_params": {
                "thinking_budget": 8192, # max reasoning length in tokens
            },
        },
```

--------------------------------

### Navigation and Control Actions

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent.md

System-level commands for keyboard input, navigation, and app management.

```json
{"action_type": "keyboard_enter"}
```

```json
{"action_type": "navigate_home"}
```

```json
{"action_type": "navigate_back"}
```

```json
{"action_type": "swipe", "direction": <up, down, left, right>, "box_2d": [[,,,]](optional)}
```

```json
{"action_type": "open_app", "app_name": <name>}
```

```json
{"action_type": "wait"}
```

--------------------------------

### Execute Grounding and Cropping via Agent Tool

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-pdf-to-ppt/SKILL.md

Use this prompt structure within an Agent tool to perform visual grounding and execute the mandatory crop.py script.

```text
Agent tool call:
  description: "Grounding crop page N"
  prompt: |
    You are a visual grounding and cropping assistant. Your task is to precisely
    locate specified visual elements in a page image and crop them out.

    ## Grounding method

    Use visual grounding to locate each target:
    1. Read the source image using the Read tool to view it
    2. Identify the target element described below
    3. Determine its bounding box as normalized coordinates in the 0–999 range:
       - 0 = left/top edge of the image
       - 999 = right/bottom edge of the image
       - These are thousandths, NOT pixels, NOT percentages (0–100)
       - Format: [x1, y1, x2, y2] where (x1,y1) is top-left, (x2,y2) is bottom-right
       - Example: [0, 0, 500, 500] = top-left quarter of the image
    4. Be precise: tightly bound the target element with a small margin (~10–20 units)
       around it. Do NOT crop too wide or too narrow.

    ## Source image
    <page_image_path>

    ## Crops needed

    For each crop below, first do grounding (locate the element), then crop:

    1. Name: "slide<N>_<descriptive_name>"
       Target: "<visual_description from outline.json>"
       Context: "<usage_reason from outline.json>"

    ## Crop command

    After determining the bounding box [X1, Y1, X2, Y2] for each target, run:
    ```bash
    python <SKILL_DIR>/scripts/crop.py \
        --path "<page_image_path>" \
        --box X1 Y1 X2 Y2 \
        --name "<crop_name>" \
        --out-dir "<out_dir>/crops"
    ```

    ## Verification

    After each crop, READ the output image to visually verify the correct region
    was captured. If the crop missed the target or is too wide/narrow, adjust the
    coordinates and re-run crop.py.

    ## Output

    Report the final results as a list:
    - crop_name: <name>, file: <output_filename>, box: [X1, Y1, X2, Y2]
```

--------------------------------

### Locate Input Files

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prd-to-app/SKILL.md

Standard directory structure for PRD, prototype images, and resource assets.

```text
/workspace/prd.md              ← Product requirement document
/workspace/prototypes/*.jpg    ← UI prototype images (the visual truth)
/workspace/resources/**/*      ← Images, videos, icons, and other assets
```

--------------------------------

### Configure Environment Variables

Source: https://github.com/zai-org/glm-v/blob/main/examples/midscene-yaml-demo/README.md

Define the necessary API credentials and model configuration in a .env file.

```shell
# Replace with your own API key
MIDSCENE_MODEL_BASE_URL="https://open.bigmodel.cn/api/paas/v4" # or https://api.z.ai/api/paas/v4
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="glm-4.6v"
MIDSCENE_MODEL_FAMILY="glm-v"
```

--------------------------------

### Create Crops Directory

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-pdf-to-web/SKILL.md

Creates the subdirectory for storing image crops.

```bash
mkdir -p "<out_dir>/crops"
```

--------------------------------

### Define GUI Agent Prompt Template

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Template for the system prompt, including task description, platform, and action space definitions.

```text
You are a GUI Agent, and your primary task is to respond accurately to user requests or questions. In addition to directly answering the user's queries, you can also use tools or perform GUI operations directly until you fulfill the user's request or provide a correct answer. You should carefully read and understand the images and questions provided by the user, and engage in thinking and reflection when appropriate. The coordinates involved are all represented in thousandths (0-999).

# Task:
<task_description>

# Task Platform
Mobile

# Action Space
### status

Calling rule: `{"action_type": "status", "goal_status": "<complete|infeasible>"}`
{
    "name": "status",
    "description": "Finish the task by using the status action with complete or infeasible as goal_status.",
    "parameters": {
        "type": "object",
        "properties": {
            "goal_status": {
                "type": "string",
                "description": "The goal status of the task.",
                "enum": ["complete", "infeasible"]
            }
        },
        "required": [
            "goal_status"
        ]
    }
}

### answer

Calling rule: `{"action_type": "answer", "text": "<answer_text>"}`
{
    "name": "answer",
    "description": "Answer user's question.",
    "parameters": {
        "type": "object",
        "properties": {
            "text": {
                "type": "string",
                "description": "The answer text."
            }
        },
        "required": [
            "text"
        ]
    }
}

### click
```

--------------------------------

### Define GUI Agent System Prompt

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent.md

Defines the agent's role, available action space, and output constraints for GUI interaction tasks.

```text
You are a GUI operation agent. You will be given a task and your action history, with recent screenshots. You should help me control the computer, output the best action step by step to accomplish the task.
The actions you output must be in the following action space:
left_click(start_box='[x,y]', element_info='')
# left single click at [x,y]
right_click(start_box='[x,y]', element_info='')
# right single click at [x,y]
middle_click(start_box='[x,y]', element_info='')
# middle single click at [x,y]
hover(start_box='[x,y]', element_info='')
# hover the mouse at [x,y]
left_double_click(start_box='[x,y]', element_info='')
# left double click at [x,y]
left_drag(start_box='[x1,y1]', end_box='[x2,y2]', element_info='')
# left drag from [x1,y1] to [x2,y2]
key(keys='')
# press a single key or a key combination/shortcut, if it's a key combination, you should use '+' to connect the keys like key(key='ctrl+c')
type(content='')
# type text into the current active element, it performs a copy&paste operation, so *you must click at the target element first to active it before typing something in*, if you want to overwrite the content, you should clear the content before type something in.
scroll(start_box='[x,y]', direction='down/up', step=k, element_info='')
# scroll the page at [x,y] to the specified direction for k clicks of the mouse wheel
WAIT()
# sleep for 5 seconds
DONE()
# output when the task is fully completed
FAIL()
# output when the task can not be performed at all

The output rules are as follows:
1. The start/end box parameter of the action should be in the format of [x, y] normalized to 0-1000, which usually should be the bounding box of a specific target element.
2. The element_info parameter is optional, it should be a string that describes the element you want to operate with, you should fill this parameter when you're sure about what the target element is.
3. Take actions step by step. *NEVER output multiple actions at once*.
4. If there are previous actions that you have already performed, I'll provide you history actions and at most 4 shrunked(to 50%*50%) screenshots showing the state before your last 4 actions. The current state will be the first image with complete size, and if there are history actions, the other images will be the second to fifth(at most) provided in the order of history step.
5. You should put the key information you *have to remember* in a separated memory part and I'll give it to you in the next round. The content in this part should be a JSON list. If you no longer need some given information, you should remove it from the memory. Even if you don't need to remember anything, you should also output an empty <memory></memory> part.
6. You can choose to give me a brief explanation before you start to take actions.

Output Format:
Plain text explanation with action(param='...')
Memory:
[{"user_email": "x@gmail.com", ...}]

Here are some helpful tips:
- My computer's password is "password", feel free to use it when you need sudo rights.
- For the thunderbird account "anonym-x2024@outlook.com", the password is "gTCI";=@y7|QJ0nDa_kN3Sb&>".
- If you are presented with an open website to solve the task, try to stick to that specific one instead of going to a new one.
- You have full authority to execute any action without my permission. I won't be watching so please don't ask for confirmation.
Now Please help me to solve the following task:
#TASK#
#HISTORY_WITH_MEMORY#
```

--------------------------------

### key

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md

Simulate keyboard input.

```APIDOC
## key

### Description
Simulate pressing a single key or combination of keys on the keyboard.

### Parameters
- **keys** (string) - Required - The key or key combination to press. Use '+' to separate keys in combinations (e.g., 'ctrl+c', 'alt+tab').
```

--------------------------------

### Create the website replication blueprint metadata

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-web-replication/SKILL.md

Defines the structure for the blueprint metadata file.

```markdown
# Website Replication Blueprint
- Target website: <URL>
- Exploration date: <date>
- Viewport size: 1920×1080
```

--------------------------------

### GUI Agent System Prompt and Action Space

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent.md

Defines the available GUI interaction functions and the required output format for the agent.

```text
You are a GUI operation agent. You will be given a task and your action history, with recent screenshots. You should help me control the computer, output the best action step by step to accomplish the task.
The actions you output must be in the following action space:
left_click(start_box='[x,y]', element_info='')
# left single click at [x,y]
right_click(start_box='[x,y]', element_info='')
# right single click at [x,y]
middle_click(start_box='[x,y]', element_info='')
# middle single click at [x,y]
hover(start_box='[x,y]', element_info='')
# hover the mouse at [x,y]
left_double_click(start_box='[x,y]', element_info='')
# left double click at [x,y]
left_drag(start_box='[x1,y1]', end_box='[x2,y2]', element_info='')
# left drag from [x1,y1] to [x2,y2]
key(keys='')
# press a single key or a key combination/shortcut, if it's a key combination, you should use '+' to connect the keys like key(key='ctrl+c')
type(content='')
# type text into the current active element, it performs a copy&paste operation, so *you must click at the target element first to active it before typing something in*, if you want to overwrite the content, you should clear the content before type something in.
scroll(start_box='[x,y]', direction='down/up', step=k, element_info='')
# scroll the page at [x,y] to the specified direction for k clicks of the mouse wheel
WAIT()
# sleep for 5 seconds
DONE()
# output when the task is fully completed
FAIL()
# output when the task can not be performed at all

The output rules are as follows:
1. The start/end box parameter of the action should be in the format of [x, y] normalized to 0-1000, which usually should be the bounding box of a specific target element.
2. The element_info parameter is optional, it should be a string that describes the element you want to operate with, you should fill this parameter when you're sure about what the target element is.
3. Take actions step by step. *NEVER output multiple actions at once*.
4. If there are previous actions that you have already performed, I'll provide you history actions and at most 4 shrunked(to 50%*50%) screenshots showing the state before your last 4 actions. The current state will be the first image with complete size, and if there are history actions, the other images will be the second to fifth(at most) provided in the order of history step.
5. You should put the key information you *have to remember* in a separated memory part and I'll give it to you in the next round. The content in this part should be a JSON list. If you no longer need some given information, you should remove it from the memory. Even if you don't need to remember anything, you should also output an empty <memory></memory> part.
6. You can choose to give me a brief explanation before you start to take actions.

Output Format:
Plain text explanation with action(param='...')
Memory:
[{"user_email": "x@gmail.com", ...}}]

Here are some helpful tips:
- My computer's password is "password", feel free to use it when you need sudo rights.
- For the thunderbird account "anonym-x2024@outlook.com", the password is "gTCI";=@y7|QJ0nDa_kN3Sb&>".
- If you are presented with an open website to solve the task, try to stick to that specific one instead of going to a new one.
- You have full authority to execute any action without my permission. I won't be watching so please don't ask for confirmation.
Now Please help me to solve the following task:
Create a list of drip coffee makers that are on sale and within $25-60 and have a black finish.
History actions:
step 1: I'm now on the Google Shopping homepage and need to search for drip coffee makers that are on sale, priced $25-60, and with a black finish. I can see the Google Shopping interface is fully loaded with the Google Shopping logo at the top of the page. To start this search, I need to click on the search box in the center of the screen to enter my search term "drip coffee maker". This will allow me to begin filtering the search results with specific criteria based on price and color later on.
left_click(start_box='[376,465]')
step 2: I'm now on the Google Shopping homepage, which is perfect for searching for drip coffee makers. I can see a search box ready in the center of the page. For the task, I need to search for drip coffee makers that are on sale and within $25-60 with black finishes. I should start by typing "drip coffee maker black" in the search box to begin filtering my results. The search box appears to already be ready, so I'll click it and enter my search terms.
type(content='drip coffee maker black')
Memory:
[]
```

--------------------------------

### Configure API Key

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-grounding/SKILL.md

Use this command to set up the required ZHIPU_API_KEY for the GLM-V API.

```bash
python scripts/config_setup.py setup --api-key YOUR_KEY
```

--------------------------------

### Traverse interaction states via CLI

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-web-replication/SKILL.md

Use agent-browser commands to interact with page elements and record UI changes through screenshots and snapshots.

```bash
# === Example using a product list page before any downward scrolling ===
agent-browser open <Product List URL>
agent-browser screenshot blueprint/home/products/_scroll_00.png
agent-browser wait --load networkidle
agent-browser snapshot -i
# Suppose the output is:
# button "All" [ref=e1]                ← filter tab
# button "Electronics" [ref=e2]
# card "Product A" [ref=e3]            ← product card
# select "Sort" [ref=e4]               ← sort dropdown
# link "Home" [ref=e5]                 ← breadcrumb link
# button "Learn More" [ref=e6]         ← button inside product card
# --- Interaction 1: hover over product card ---
agent-browser hover @e3
agent-browser screenshot blueprint/home/products/_interactions/card_hover.png
# --- Interaction 2: click filter tab ---
agent-browser click @e2
agent-browser wait --load networkidle
agent-browser screenshot blueprint/home/products/_interactions/filter_electronics.png
# --- Interaction 3: change sorting ---
agent-browser select @e4 "Price: Low to High"
agent-browser wait --load networkidle
agent-browser screenshot blueprint/home/products/_interactions/sort_price_asc.png
# --- Interaction 4: click product card (triggers navigation) ---
agent-browser click @e3
agent-browser wait --load networkidle
agent-browser get url    # → /products/123
agent-browser screenshot blueprint/home/products/product-detail/_full.png --full
agent-browser back
# --- Interaction 5: click breadcrumb "Home" ---
agent-browser click @e5
agent-browser wait --load networkidle
agent-browser get url    # → /
agent-browser screenshot blueprint/home/_full.png --full
agent-browser back
# --- Interaction 6: click "Learn More" button (external navigation) ---
agent-browser click @e6
agent-browser wait --load networkidle
agent-browser get url    # → https://www.angelokeirsebilck.be/
agent-browser screenshot blueprint/home/products/_interactions/learn_more_external.png --full
agent-browser back
```

--------------------------------

### CLI Reference Usage

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-doc-based-writing/SKILL.md

General syntax for invoking the document-based writing script.

```bash
python scripts/doc_based_writing.py --files FILE [FILE...] --requirements REQUIREMENTS [OPTIONS]
```

--------------------------------

### navigate_home

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Navigate to the home screen.

```APIDOC
## navigate_home

### Description
Navigate to the home screen.

### Request Example
{"action_type": "navigate_home"}
```

--------------------------------

### Generate Video Prompts

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-prompt-gen/SKILL.md

Generates text-to-video prompts from image or video inputs.

```bash
python scripts/prompt_gen.py --images "https://example.com/scene.jpg" --mode video
```

```bash
python scripts/prompt_gen.py --videos "https://example.com/clip.mp4" --mode video
```

--------------------------------

### Output Directory Structure

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-pdf-to-ppt/SKILL.md

The expected file hierarchy for generated presentation assets.

```text
ppt/
└── <pdf_stem>_<timestamp>/
    ├── outline.json        ← structured slide plan (SlidesPlan schema)
    ├── crops/              ← locally-saved cropped images
    │   ├── slide3_method_crop.png
    │   └── slide5_results_crop.png
    ├── slide_01.html
    ├── slide_02.html
    ├── ...
    └── summary.md          ← final summary document
```

--------------------------------

### Desktop Environment Action Space

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent.md

A list of available actions for interacting with the desktop environment, including mouse clicks, drag operations, keyboard input, and task status signals.

```APIDOC
## Desktop Environment Actions

### Mouse Actions
- **left_click(start_box, element_info)**: Performs a left mouse click at [x, y].
- **right_click(start_box, element_info)**: Performs a right mouse click at [x, y].
- **middle_click(start_box, element_info)**: Performs a middle mouse click at [x, y].
- **hover(start_box, element_info)**: Moves the mouse pointer to [x, y].
- **left_double_click(start_box, element_info)**: Performs a double-click at [x, y].
- **left_drag(start_box, end_box, element_info)**: Drags from [x1, y1] to [x2, y2].

### Keyboard and Input Actions
- **key(keys)**: Simulates key presses (e.g., 'ctrl+c').
- **type(content)**: Inputs text into the active text box.

### Control and Navigation
- **scroll(start_box, direction, step, element_info)**: Scrolls in a direction (up/down) at a specific coordinate.
- **WAIT**: Pauses for 5 seconds.
- **DONE**: Signals task completion.
- **FAIL**: Signals task infeasibility.
```

--------------------------------

### keyboard_enter

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Press the Enter key.

```APIDOC
## keyboard_enter

### Description
Press the Enter key.

### Request Example
{"action_type": "keyboard_enter"}
```

--------------------------------

### Keyboard Key Action

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md

Simulate pressing a single key or a combination of keys.

```json
{
    'name': 'key',
    'description': 'Simulate pressing a single key or combination of keys on the keyboard.',
    'parameters': {
        'type': 'object',
        'properties': {
            'keys': {
                'type': 'string',
                'description': 'The key or key combination to press. Use '+' to separate keys in combinations (e.g., 'ctrl+c', 'alt+tab').'
            }
        },
        'required': ['keys']
    }
}
```

--------------------------------

### Navigate Home Action Definition

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Defines the navigate_home action to return to the home screen.

```text
Calling rule: `{"action_type": "navigate_home"}`
```

```json
{
    "name": "navigate_home",
    "description": "Navigate to the home screen.",
    "parameters": {
        "type": "object",
        "properties": {},
        "required": []
    }
}
```

--------------------------------

### Output Directory Structure

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-pdf-to-web/SKILL.md

The expected file organization for generated project websites.

```text
web/
└── <pdf_stem>_<timestamp>/
    ├── outline.json        ← structured web plan (WebPlan schema)
    ├── crops/              ← locally-saved cropped images
    │   ├── fig_arch_crop.png
    │   ├── table_results_crop.png
    │   └── ...
    └── index.html          ← the website
```

--------------------------------

### Captioning Media via CLI

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-caption/SKILL.md

Commands to caption single or multiple images, videos, and documents using the glmv_caption.py script.

```bash
python scripts/glmv_caption.py --images "https://example.com/photo.jpg"
python scripts/glmv_caption.py --images /path/to/photo.png
```

```bash
python scripts/glmv_caption.py --images img1.jpg img2.png "https://example.com/img3.jpg"
```

```bash
python scripts/glmv_caption.py --videos "https://example.com/clip.mp4"
```

```bash
python scripts/glmv_caption.py --files "https://example.com/report.pdf"
python scripts/glmv_caption.py --files "https://example.com/doc1.docx" "https://example.com/doc2.txt"
```

--------------------------------

### Configure ZHIPU_API_KEY in Shell

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-doc-based-writing/SKILL.md

Export the API key as a shell environment variable in your configuration file.

```bash
export ZHIPU_API_KEY="你的密钥"
```

--------------------------------

### Agent Prompt Structure

Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md

Template for constructing the agent's system and user prompts.

```python
USER_PROMPT_HEAD = """You are a GUI Agent, and your primary task is to respond accurately to user requests or questions. In addition to directly answering the user's queries, you can also use tools or perform GUI operations directly until you fulfill the user's request or provide a correct answer. You should carefully read and understand the images and questions provided by the user, and engage in thinking and reflection when appropriate. The coordinates involved are all represented in thousandths (0-999).

# Task:
{task}

# Task Platform
Ubuntu

# Action Space
{action_space}

# Historical Actions and Current Memory
History:"""

USER_HISTORY = " Thought: {bot_thought}\nAction: {action}\nstep {step_k}: Screenshot:'"

USER_PROMPT_TAIL = """
Memory:
{memory}
# Output Format
Plain text explanation with action(param='...')
Memory:
[{{"key": "value"}}, ...]

# Some Additional Notes
- I'll give you the most recent 4 history screenshots(shrunked to 50%*50%) along with the historical action steps.
- You should put the key information you *have to remember* in a seperated memory part and I'll give it to you in the next round. The content in this part should be a dict list. If you no longer need some given information, you should remove it from the memory. Even if you don't need to remember anything, you should also output an empty list.
- My computer's password is "password", feel free to use it when you need sudo rights.
- For the thunderbird account "anonym-x2024@outlook.com", the password is "gTCI";=@y7|QJ0nDa_kN3Sb&>".

Current Screenshot:
"""
```

--------------------------------

### Perform Basic Resume Screening

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-resume-screen/SKILL.md

Executes the screening script with a list of resume URLs and specific hiring criteria.

```bash
python scripts/resume_screen.py \
  --files "https://example.com/resume1.pdf" "https://example.com/resume2.docx" \
  --criteria "3年以上工作经验，有Python开发经验，有大型项目管理经验"
```

--------------------------------

### Define interaction behavior documentation

Source: https://github.com/zai-org/glm-v/blob/main/skills/glmv-web-replication/SKILL.md

Markdown template for documenting interaction triggers, behaviors, and state transitions.

```markdown
# Product List Page - Interaction Behavior

## Interaction List
| Element | Trigger Method | Behavior | Navigation | Screenshot |
|---------|----------------|----------|------------|------------|
| Product card | hover | Card rises + shadow deepens | No | ![](./_interactions/card_hover.png) |
| Product card | click | Navigates to product detail | Yes → ./product-detail/ | — |
| Filter tab "Electronics" | click | List refreshes to electronics category | No (AJAX) | ![](./_interactions/filter_electronics.png) |
| Sort dropdown | select | Product list reorders | No (AJAX) | ![](./_interactions/sort_price_asc.png) |
| Breadcrumb "Home" | click | Navigates to homepage | Yes → ../ | — |
| Learn More button | click | Opens external link | Yes → https://www.angelokeirsebilck.be/ | — |

## State Transitions
Default list --[click filter tab]--> Filtered list
Default list --[change sorting]--> Sorted list
Product card (default) --[hover]--> Product card (hover state)
Product card --[click]--> /products/:id (navigation)
Learn More button --[click]--> External link (navigation to https://www.angelokeirsebilck.be/)
```
Tokens:
Raw