### Example Action: Left Click Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md An example of a left-click action with specified start box coordinates and element information, as used in the GIMP interface. ```Python left_click(start_box='[168, 213]', element_info='Transform option') ``` -------------------------------- ### Environment Installation Source: https://github.com/zai-org/glm-v/blob/main/README.md Installs necessary dependencies for SGLang and transformers, and provides specific installation commands for vLLM with a preview version of transformers. ```bash pip install -r requirements.txt ``` ```bash pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly pip install transformers-v4.55.0-GLM-4.5V-preview ``` -------------------------------- ### Install Dependencies Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README_zh.md Installs project dependencies using pnpm, the package manager specified for this project. ```bash pnpm install ``` -------------------------------- ### Configure Reward System Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README_zh.md Copies the example configuration template to a new file for customization. ```bash cp examples/configs/example.yaml.template examples/configs/example.yaml ``` -------------------------------- ### Initial Action Example Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md Demonstrates the first action taken by the GUI Agent to initiate a search on GitHub, specifically clicking on the search bar element. ```APIDOC 0.Thought:The task is to find a specific GitHub repository. I need to use the search bar to start the search. The search bar is labeled 9. Action:Click [9] Observation:Success ``` -------------------------------- ### Configure Reward System Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md Copies the example configuration template and prepares it for customization. Users should edit the copied file to suit their specific needs. ```bash cp examples/configs/example.yaml.template examples/configs/example.yaml ``` -------------------------------- ### Initial Action Example Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md Demonstrates the first action taken by the GUI Agent to initiate a search on GitHub, involving clicking on a search bar element. ```APIDOC 0.Thought:The task is to find a specific GitHub repository. I need to use the search bar to start the search. The search bar is labeled 9. Action:Click [9] Observation:Success ``` -------------------------------- ### GUI Agent Message Formatting Example Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md An example of how to structure a message for a GUI agent when historical steps are present. It includes the agent's role, the task, platform, and the available action space with detailed API documentation for each action. ```python [ {'role': 'user', 'content': [ {'type': 'text', 'text': 'You are a GUI Agent, and your primary task is to respond accurately to user requests or questions. In addition to directly answering the user\'s queries, you can also use tools or perform GUI operations directly until you fulfill the user\'s request or provide a correct answer. You should carefully read and understand the images and questions provided by the user, and engage in thinking and reflection when appropriate. The coordinates involved are all represented in thousandths (0-999).\n\n# Task:\nPlease rotate my figure to mirror it horizontally.\n\n# Task Platform\nUbuntu\n\n# Action Space\n\n### {left,right,middle}_click\n\nCall rule: `{left,right,middle}_click(start_box=\'[x,y]\', element_info=\'\')`\n{\n \'name\': [\'left_click\', \'right_click\', \'middle_click\'],\n \'description\': \'Perform a left/right/middle mouse click at the specified coordinates on the screen.\',\n \'parameters\': {\n \'type\': \'object\',\n \'properties\': {\n \'start_box\': {\n \'type\': \'array\',\n \'items\': {\n \'type\': \'integer\'\n },\n \'description\': \'Coordinates [x,y] where to perform the click, normalized to 0-999 range.\'\n },\n \'element_info\': {\n \'type\': \'string\',\n \'description\': \'Optional text description of the UI element being clicked.\'\n }\n },\n \'required\': [\'start_box\']\n }\n}\n\n### hover\n\nCall rule: `hover(start_box=\'[x,y]\', element_info=\'\')`\n{\n \'name\': \'hover\',\n \'description\': \'Move the mouse pointer to the specified coordinates without performing any click action.\',\n \'parameters\': {\n \'type\': \'object\',\n \'properties\': {\n \'start_box\': {\n \'type\': \'array\',\n \'items\': {\n \'type\': \'integer\'\n },\n \'description\': \'Coordinates [x,y] where to move the mouse pointer, normalized to 0-999 range.\'\n },\n \'element_info\': {\n \'type\': \'string\',\n \'description\': \'Optional text description of the UI element being hovered over.\'\n }\n },\n \'required\': [\'start_box\']\n }\n}\n\n### left_double_click\n\nCall rule: `left_double_click(start_box=\'[x,y]\', element_info=\'\')`\n{\n \'name\': \'left_double_click\',\n \'description\': \'Perform a left mouse double-click at the specified coordinates on the screen.\',\n \'parameters\': {\n \'type\': \'object\',\n \'properties\': {\n \'start_box\': {\n \'type\': \'array\',\n \'items\': {\n \'type\': \'integer\'\n },\n \'description\': \'Coordinates [x,y] where to perform the double-click, normalized to 0-999 range.\'\n },\n \'element_info\': {\n \'type\': \'string\',\n \'description\': \'Optional text description of the UI element being double-clicked.\'\n }\n },\n \'required\': [\'start_box\']\n }\n}\n\n### left_drag\n\nCall rule: `left_drag(start_box=\'[x1,y1]\', end_box=\'[x2,y2]\', element_info=\'\')`\n{\n \'name\': \'left_drag\',\n \'description\': \'Drag the mouse from starting coordinates to ending coordinates while holding the left mouse button.\',\n \'parameters\': {\n \'type\': \'object\',\n \'properties\': {\n \'start_box\': {\n \'type\': \'array\',\n \'items\': {\n \'type\': \'integer\'\n },\n \'description\': \'Starting coordinates [x1,y1] for the drag operation, normalized to 0-999 range.\'\n },\n \'end_box\': {\n \'type\': \'array\',\n \'items\': {\n \'type\': \'intege'}] } ] ``` -------------------------------- ### Environment Installation for SGLang, Transformers, and vLLM Source: https://github.com/zai-org/glm-v/blob/main/README_zh.md Provides instructions for installing necessary dependencies for the project. This includes installing requirements from `requirements.txt` for SGLang and Transformers, and specific commands for installing vLLM and a preview version of transformers for GLM-4.5V. ```Shell 对于`SGLang` 和 `transformers`，请按照`requirements.txt` 中的依赖进行安装。 ```shell pip install -r requirements.txt ``` 对于`vLLM`，请使用以下命令安装： ```shell pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly pip install transformers-v4.55.0-GLM-4.5V-preview ``` ``` -------------------------------- ### Example Math Verifier Configuration Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md An example YAML configuration snippet for the math verifier, demonstrating how to enable LLM judge fallback and specify the LLM judge URL. ```yaml reward_configs: math_verifier_config: verifier_type: "math" enable_llm_judge_fallback: true llm_judge_url: - "https://open.bigmodel.cn/api/paas/v4/chat/completions" ``` -------------------------------- ### GLM-4.5V Grounding Prompt Examples Source: https://github.com/zai-org/glm-v/blob/main/README.md Examples of query prompts for GLM-4.5V to locate objects in an image and return their bounding boxes, supporting complex descriptions and specified output formats. ```text - Help me to locate in the image and give me its bounding boxes. - Please pinpoint the bounding box [[x1,y1,x2,y2], …] in the image as per the given description. ``` -------------------------------- ### Install VLM Reward System Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README_zh.md Installs the VLM Reward System package in editable mode. ```bash pip install -e . ``` -------------------------------- ### Run in Development Mode Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README_zh.md Starts the application in development mode, allowing for live reloading and debugging. ```bash pnpm dev ``` -------------------------------- ### Install VLM Reward System Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md Installs the VLM Reward System package using pip. This command should be run in your terminal. ```bash pip install -e . ``` -------------------------------- ### Build without Packaging Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README_zh.md Builds the application artifacts without packaging them into an installer. ```bash pnpm build:unpack ``` -------------------------------- ### Run Reward System Demo Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md Executes the reward system demo script to showcase its functionality. This requires the system to be installed and configured. ```bash python examples/reward_system_demo.py ``` -------------------------------- ### Development Commands Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README.md Essential commands for managing the project during development, including dependency installation, running in development mode, type checking, linting, formatting, and building. ```bash # Install dependencies pnpm install # Development mode pnpm dev # Type checking pnpm typecheck # Lint pnpm lint # Format code pnpm format # Rebuild native modules pnpm rebuild # Build the project pnpm build # Build with type checking pnpm build:with-typecheck ``` -------------------------------- ### Google Shopping Search Example Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent_zh.md Demonstrates a step-by-step process for searching on Google Shopping for drip coffee makers with specific criteria (on sale, price range, black finish). ```Python step 1: I'm now on the Google Shopping homepage and need to search for drip coffee makers that are on sale, priced $25-60, and with a black finish. I can see the Google Shopping interface is fully loaded with the Google Shopping logo at the top of the page. To start this search, I need to click on the search box in the center of the screen to enter my search term "drip coffee maker". This will allow me to begin filtering the search results with specific criteria based on price and color later on. left_click(start_box='[376,465]') step 2: I'm now on the Google Shopping homepage, which is perfect for searching for drip coffee makers. I can see a search box ready in the center of the page. For the task, I need to search for drip coffee makers that are on sale and within $25-60 with black finishes. I should start by typing "drip coffee maker black" in the search box to begin filtering my results. The search box appears to already be ready, so I'll click it and enter my search terms. type(content='drip coffee maker black') Memory: [] ``` -------------------------------- ### Open Application Action Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md Opens a specified application. If the app is not installed, no action will occur. A list of supported apps is provided. ```APIDOC open_app: description: Open an app (nothing will happen if the app is not installed). parameters: app_name: The name of the app to open. Supported apps: Google Chrome, Settings, Camera, Audio Recorder, Clock, Contacts, Files, Markor, Simple SMS Messenger, Simple Calendar Pro, Simple Gallery Pro, Simple Draw Pro, Pro Expense, Broccoli, OsmAnd, Tasks, Open Tracks Sports Tracker, Joplin, VLC, Retro Music. calling_rule: {"action_type": "open_app", "app_name": ""} ``` -------------------------------- ### Initialize and Use Reward System Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README_zh.md Demonstrates how to initialize the RewardSystem with a configuration file and get reward scores for model responses. ```python from glmv_reward import RewardSystem # Initialize reward system reward_system = RewardSystem("examples/configs/example.yaml") # Evaluate model response rewards = reward_system.get_reward( prompts=["15 + 27 等于多少？"], answers=["15 + 27 = 42<|begin_of_box|>42<|end_of_box|>"], gt_answers=["15 + 27 = 42<|begin_of_box|>42<|end_of_box|>"], datasources=["math"] ) # Use rewards in reinforcement learning training print(f"Reward: {rewards[0]}") # Output: 1.0 (Correct answer) ``` -------------------------------- ### GIMP Image Transformation Example Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md Demonstrates a sequence of actions to perform a horizontal flip on an image using GIMP, involving clicking through menu options. ```Python Thought: I can see the Image menu is open and displaying various options. I can see there's a "Transform" option in the menu, which is likely where I'll find the horizontal mirroring functionality. I need to click on "Transform" to access the transformation options that include horizontal flipping. Action: left_click(start_box='[162, 212]', element_info='Transform option') ``` ```Python Thought: I can see the Transform submenu is open, but I don't see the horizontal flip option that I need. The Transform submenu appears to have more options that aren't currently visible. I need to look for the "Flip Horizontally" option, which is typically found within the Transform menu in GIMP. Let me click on the Transform option to see if there are additional submenu options available. Action: left_click(start_box='[168, 213]', element_info='Transform option') ``` ```Python Thought: Perfect! I can see the Transform submenu is now open and displaying the transformation options I need. I can clearly see the "Flip Horizontally" option, which is exactly what I need to mirror the figure horizontally as requested. I'll click on this option to apply the horizontal flip to the image. Action: left_click(start_box='[320, 211]', element_info='Flip Horizontally option') ``` ```Python Thought: I can see the Transform submenu is open in GIMP with the "Flip Horizontally" option visible. The cursor appears to be hovering over this option, but I need to click on it to actually apply the horizontal flip transformation to the image. I'll click on the "Flip Horizontally" option to mirror the figure as requested. Action: left_click(start_box='[339, 211]', element_info='Flip Horizontally option') ``` -------------------------------- ### Web Browsing and Action Execution Best Practices Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md Outlines crucial guidelines for web browsing and action execution, including inputting text directly, distinguishing textboxes from buttons, executing single actions, avoiding repetition, and using the 'ANSWER' action appropriately. It also covers interacting only with labeled elements and handling loading states. ```English Action guidelines * 1) To input text, NO need to click textbox first, directly type content. After typing, the system automatically hits `ENTER` key. Sometimes you should click the search button to apply search filters. Try to use simple language when searching. 2) You must Distinguish between textbox and search button, don't type content into the button! If no textbox is found, you may need to click the search button first before the textbox is displayed. 3) Execute only one action per iteration. 4) STRICTLY Avoid repeating the same action if the webpage remains unchanged. You may have selected the wrong web element or numerical label. Continuous use of the Wait is also NOT allowed. 5) When a complex Task involves multiple questions or steps, select "ANSWER" only at the very end, after addressing all of these questions (steps). Flexibly combine your own abilities with the information in the web page. Double check the formatting requirements in the task when ANSWER. 6) You can only interact with web elements in the screenshot that have s numerical label.Before giving the action, double-check that the numerical label appears on the screen. 7) If any web elements with numerical labels in the screenshot have not finished loading, you need to wait for them to load completely. * Web Browsing Guidelines * 1) Don't interact with useless web elements like Login, Sign-in, donation that appear in Webpages. Pay attention to Key Web Elements like search textbox and menu. 2) Vsit video websites like YouTube is allowed BUT you can't play videos. Clicking to download PDF is allowed and will be analyzed by the Assistant API. 3) Focus on the numerical labels in the TOP LEFT corner of each rectangle (element). Ensure you don't mix them up with other numbers (e.g. Calendar) on the page. 4) Focus on the date in task, you must look for results that match the date. It may be necessary to find the correct year, month and day at calendar. 5）During the process of browsing web page content, to ensure the complete acquisition of the target content, it may be necessary to scroll down the page until confirming the appearance of the end marker for the target content. For example, when new webpage information appears, or the webpage scroll bar has reached the bottom, etc. 6) When you work on the github website, use "Key; [Return]" to start a search. 7) Try your best to find the answer that best fits the task, if any situations that do not meet the task requirements occur during the task, correct the mistakes promptly. ``` -------------------------------- ### Build Application for All Platforms Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README_zh.md Builds the application for all supported platforms. ```bash pnpm build ``` -------------------------------- ### Math Verifier Configuration Example Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README_zh.md An example YAML configuration for the math verifier, enabling LLM judge fallback. ```yaml reward_configs: math_verifier_config: verifier_type: "math" enable_llm_judge_fallback: true llm_judge_url: - "https://open.bigmodel.cn/api/paas/v4/chat/completions" ``` -------------------------------- ### Project Build Commands Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README.md Commands for building the Electron application for various platforms and configurations. Includes building for all platforms, specific platforms, and without packaging. ```bash # Build all platforms pnpm build # Build Windows pnpm build:win # Build macOS pnpm build:mac # Build Linux pnpm build:linux # Build without packaging pnpm build:unpack ``` -------------------------------- ### Example Action: Left Click on Flip Horizontally Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md An example of a left-click action targeting the 'Flip Horizontally' option within a GIMP Transform submenu. ```Python left_click(start_box='[320, 211]', element_info='Flip Horizontally option') ``` -------------------------------- ### Project Structure Overview Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README.md Illustrates the directory structure of the VLM Chat Helper project, detailing the organization of main process code, renderer process code, preload scripts, and build assets. ```typescript vlm-chat-helper/ ├── src/ │ ├── main/ # Main process code │ │ ├── index.ts # Main process entry │ │ ├── modules/ # Feature modules │ │ │ ├── windowManager.ts # Window management │ │ │ ├── shortcutManager.ts # Global hotkeys │ │ │ ├── recordingManager.ts # Screen recording │ │ │ └── ipcHandlers.ts # IPC handlers │ │ ├── services/ # Service layer │ │ │ └── database.ts # Database service │ │ └── utils/ # Utilities │ ├── preload/ # Preload scripts │ │ └── index.ts # Preload entry │ └── renderer/ # Renderer process code │ └── src/ │ ├── App.vue # Root component │ ├── main.ts # Renderer entry │ ├── components/ # Shared components │ ├── views/ # Pages / views │ ├── stores/ # State management │ ├── composables/ # Composable utilities │ └── utils/ # Utilities ├── build/ # Build assets ├── resources/ # App resources ├── package.json # Project config ├── electron.vite.config.ts # Electron Vite config └── electron-builder.yml # Packaging config ``` -------------------------------- ### Mouse Left Drag Action Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md Simulates dragging the mouse from a starting point to an ending point while holding the left mouse button. Requires both start and end coordinates, and optionally an element description. ```APIDOC left_drag(start_box='[x1,y1]', end_box='[x2,y2]', element_info='') - Drag the mouse from starting coordinates to ending coordinates while holding the left mouse button. - Parameters: - start_box (array[integer]): Starting coordinates [x1,y1] for the drag operation, normalized to 0-999 range. - end_box (array[integer]): Ending coordinates [x2,y2] for the drag operation, normalized to 0-999 range. - element_info (string, optional): Optional text description of the UI element being dragged. ``` -------------------------------- ### Action Handling and Navigation Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md Details the usage of various actions like `input_text`, `swipe`, and `navigate_back`. It includes tips for effective screen navigation and keyboard management. ```English Use `input_text` to type into fields, remembering to clear default text first. Use `swipe` with directions like 'up', 'down', 'left', 'right' to reveal content. If one direction doesn't work, try the opposite. Use `navigate_back` to close the soft keyboard. ``` -------------------------------- ### Run Tests Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README_zh.md Runs the test suite to verify the installation and functionality of the reward system. ```bash pytest tests/ ``` -------------------------------- ### Build Application for Windows Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README_zh.md Builds the application specifically for Windows. ```bash pnpm build:win ``` -------------------------------- ### Action Handling and Navigation Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md Details the usage of specific actions like `input_text`, `swipe`, and `navigate_back` for user interaction and system navigation. It includes tips for effective swipe gestures and keyboard management. ```General Usage Use `input_text` for typing, clearing default text first. Use `swipe` to explore content; direction is opposite to content movement (e.g., swipe 'up' to see content at the bottom). Use `navigate_back` to close the soft keyboard. ``` -------------------------------- ### Open App Action Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md Opens a specified application. If the app is not installed, no action will be taken. ```APIDOC open_app: Open an app (nothing will happen if the app is not installed). Parameters: - app_name: The name of the app to open. Supported apps: ``` -------------------------------- ### App-Specific Instructions Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md Provides guidance for interacting with specific applications, including the Files app (view modes, search) and the Markor app (save button, welcome screen). ```General Usage Files app: Switch view or use search for incomplete file names. Markor app: Save button is a floppy disk icon in the top toolbar. Skip welcome screen by tapping 'right arrow' and 'DONE'. ``` -------------------------------- ### Web Browsing Guidelines Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md Outlines rules for browsing web pages, including element selection, website restrictions, and content acquisition. ```APIDOC Web Browsing: - Useless Elements: Do not interact with non-essential elements like 'Login', 'Sign-in', or 'donation' links. - Media Websites: Visiting video websites like YouTube is allowed, but playing videos is prohibited. Downloading PDFs is permitted and will be analyzed. - Numerical Label Focus: Concentrate on the numerical labels in the TOP LEFT corner of element rectangles. Avoid confusion with other numbers on the page (e.g., calendar dates). - Date Matching: Focus on the task's specified date and look for matching results. This may involve finding the correct year, month, and day on a calendar. - Content Acquisition: Scroll down the page to ensure complete acquisition of target content. Continue scrolling until an end marker appears, new information loads, or the scroll bar reaches the bottom. - GitHub Search: On GitHub, use 'Key; [Return]' to initiate a search. - Task Adherence: Strive to find the best answer fitting the task requirements. Correct any mistakes promptly if task requirements are not met. ``` -------------------------------- ### General Project Guidelines Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md Outlines general principles for completing tasks, including choosing the easiest method, retrying operations, and handling questions or chat messages. ```English Choose the easiest method to complete a task. Retry operations if they don't work as expected. For questions/chat, use the `answer` action before finishing. If a desired state is already achieved, the task can be completed. ``` -------------------------------- ### Run Tests Source: https://github.com/zai-org/glm-v/blob/main/glmv_reward/README.md Executes the test suite for the VLM Reward System using pytest to verify the installation and core functionalities. ```bash pytest tests/ ``` -------------------------------- ### Web Browsing and Element Interaction Guidelines Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md Provides crucial guidelines for interacting with web pages, focusing on using numerical labels from screenshots, distinguishing between textboxes and buttons, executing single actions, avoiding repetition, and handling complex tasks. It also covers guidelines for visiting websites, handling dates, scrolling for content, and specific instructions for GitHub. ```APIDOC Web Interaction Guidelines: - Use numerical labels from screenshots for interaction. - Do not click textboxes before typing; system auto-hits 'ENTER'. - Distinguish textboxes from buttons; do not type into buttons. - Execute only one action per iteration. - Avoid repeating the same action if the webpage is unchanged. - Select 'ANSWER' only at the very end of a multi-step task. - Interact only with web elements that have numerical labels. - Wait for web elements with numerical labels to load completely. Web Browsing Guidelines: - Avoid interacting with useless elements (e.g., Login, Sign-in, donation). - Focus on key elements like search textboxes and menus. - Visiting video websites (e.g., YouTube) is allowed, but videos cannot be played. - Clicking to download PDFs is allowed. - Focus on numerical labels in the TOP LEFT corner of elements. - Match results to the date specified in the task, including year, month, and day. - Scroll down to ensure complete acquisition of target content. - For GitHub, use "Key; [Return]" to start a search. - Correct mistakes promptly if task requirements are not met. ``` -------------------------------- ### APIDOC: Drag and Drop Actions Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md Defines the parameters for drag and drop operations, including start and end coordinates and optional element descriptions. ```APIDOC drag_and_drop: description: Simulate a drag and drop operation. parameters: type: object properties: start_box: type: array items: type: integer description: Starting coordinates [x,y] for the drag operation, normalized to 0-999 range. end_box: type: array items: type: integer description: Ending coordinates [x2,y2] for the drag operation, normalized to 0-999 range. element_info: type: string description: Optional text description of the UI element being dragged. required: - start_box - end_box ``` -------------------------------- ### Web Environment Prompt Template Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent.md Provides the structure for prompts used when interacting with the web environment, including task details, web context, time, memory, and previous actions. ```APIDOC # Prompt Template You are a GUI Agent, and your primary task is to respond accurately to user requests or questions. In addition to directly answering the user's queries, you can also use tools or perform GUI operations directly until you fulfill the user's request or provide a correct answer. You should carefully read and understand the images and questions provided by the user, and engage in thinking and reflection when appropriate. The coordinates involved are all represented in thousandths (0-999). # Task: {TASK}, Please interact with {Web} and get the answer. The current time in Beijing is {Time}. # Task Platform Web # Memory: {Memory} # Previous Actions: {PREVIOUS_ACTIONS} # Current Screenshot: {web_text} ``` -------------------------------- ### GUI Agent Prompt Template Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-45v/agent_zh.md A template for constructing prompts for GUI agents, including placeholders for task details, web environment, time, memory, previous actions, and web text, to be used with a screenshot containing numerical labels. ```APIDOC You are a GUI Agent, and your primary task is to respond accurately to user requests or questions. In addition to directly answering the user's queries, you can also use tools or perform GUI operations directly until you fulfill the user's request or provide a correct answer. You should carefully read and understand the images and questions provided by the user, and engage in thinking and reflection when appropriate. The coordinates involved are all represented in thousandths (0-999). # Task: {TASK}, Please interact with {Web} and get the answer. The current time in Beijing is {Time}. # Task Platform Web # Action Space Action should STRICTLY follow the format: - Click [Numerical_Label]. For example the Action "Click [1]" means clicking on the web element with a Numerical_Label of "1". - Type [Numerical_Label]; [The input content]. For example the Action "Type [2]; [5$]" means typing "5$" in the web element with a Numerical_Label of "2". - Scroll [Numerical_Label or WINDOW]; [up or down]. For example the Action "Scroll [6]; [up]" means scrolling up in the web element with a Numerical_Label of "6". - Wait. For example the Action "Wait" means waiting for 5 seconds. - GoBack. For example the Action "GoBack" means going back to the previous webpage. - Bing. For example the Action "Bing" means jumping to the Bing search page. - ANSWER; The content of the answer. For example the Action "ANSWER; Guatemala" means answering the task with "Guatemala". - Key; [The key name]. For example the Action "Key; [Return]" means pressing the Enter key ``` -------------------------------- ### Build Application for Linux Source: https://github.com/zai-org/glm-v/blob/main/examples/vlm-helper/README_zh.md Builds the application specifically for Linux. ```bash pnpm build:linux ``` -------------------------------- ### Web Interaction Example Source: https://github.com/zai-org/glm-v/blob/main/examples/gui-agent/glm-41v/agent_zh.md Demonstrates a typical interaction flow on a webpage, including searching for content and updating memory. It follows a specific Thought-Action-Memory_Updated format. ```English Thought: I'm on the homepage of Allrecipes.com. I need to search for "vegetarian lasagna" to find recipes that meet my criteria. I'll start by typing the search query directly into the search box since element [1] is a textbox. Action:Type [1]; [Vegetarian Lasagna] Observation:Success D. The "Memory" only stores the information obtained from the web page that is relevant to the task, and the "Memory" is strictly in JSON format. For example: {"user_email_address": "test@163.com", "user_email_password": "123456", "jack_email_address": "jack@163.com"}. The "Memory" does not include future plans, descriptions of current actions, or other reflective content; it only records visual information that is relevant to the task obtained from the screenshot. The "Memory" in the current step as follow: Memory:{} E. I've provided the tag name of each element and the text it contains (if text exists). Note that