CogAgent

https://github.com/zai-org/cogagent
Admin
CogAgent is an open-sourced vision-language model that enables GUI agents to autonomously interact...
Tokens:18,399
Snippets:173
Trust Score:7.8
License:Apache-2.0
Update:5 months ago
Show doc for...
Tokens:
Raw
### Training Data Format Example

Source: https://context7.com/zai-org/cogagent/llms.txt

Example of a training data entry in JSONL format, including user messages with content and optional images, and assistant responses.

```json
{
  "messages": [
    {
      "role": "user",
      "content": "Task: Search for 'CogAgent'\nHistory steps: \n(Platform: Mac)\n(Answer in Action-Operation format.)",
      "image": null
    },
    {
      "role": "assistant",
      "content": "Action: Click on the search box.\nGrounded Operation: CLICK(box=[[352,102,786,139]], element_type='Text input', element_info='Search')",
      "image": "/path/to/screenshot.png"
    }
  ]
}
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zai-org/cogagent/blob/main/app/README.md

Install all necessary project dependencies listed in the requirements.txt file. This should be done after ensuring tkinter is installed.

```shell
pip install -r requirements.txt
```

--------------------------------

### Run Web Demo

Source: https://github.com/zai-org/cogagent/blob/main/README.md

Start the web demo for CogAgent, enabling continuous image uploads for interactive inference. Configure host, port, model directory, format key, platform, and output directory.

```shell
python inference/web_demo.py --host 0.0.0.0 --port 7860 --model_dir THUDM/cogagent-9b-20241220 --format_key status_action_op_sensitive --platform "Mac" --output_dir ./results
```

--------------------------------

### Client Usage with OpenAI SDK

Source: https://context7.com/zai-org/cogagent/llms.txt

Example of how to interact with the OpenAI-compatible API server using the OpenAI SDK. Includes image encoding and constructing chat completion requests.

```python
import base64
from openai import OpenAI

def encode_image(image_path: str) -> str:
    """Encode image to base64 string."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

client = OpenAI(
    api_key="your-api-key",  # Can be any string for local deployment
    base_url="http://localhost:8000/v1"
)

# Prepare the prompt
task = "Please mark all my emails as read"
platform_str = "(Platform: Mac)\n"
format_str = "(Answer in Action-Operation-Sensitive format.)\n"
query = f"Task: {task}\nHistory steps: \n{platform_str}{format_str}"

# Encode screenshot
image_base64 = encode_image("screenshot.png")

response = client.chat.completions.create(
    model="CogAgent",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": query},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}
                }
            ]
        }
    ],
    max_tokens=1024,
    temperature=0.6,
    top_p=0.8
)

print(response.choices[0].message.content)
```

--------------------------------

### Example of a Constructed Python Prompt String

Source: https://github.com/zai-org/cogagent/blob/main/README_zh.md

This is an example of a fully constructed Python prompt string, showing the expected format for task, history, platform, and output format.

```python
"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
```

--------------------------------

### Install Python Tkinter and Dependencies

Source: https://github.com/zai-org/cogagent/blob/main/app/README_en.md

Installs the tkinter library using Homebrew and then installs project dependencies from requirements.txt. Ensure you have Homebrew installed for this command to work.

```shell
brew install python-tk
pip install -r requirements.txt
```

--------------------------------

### Start vLLM OpenAI-Compatible Server

Source: https://context7.com/zai-org/cogagent/llms.txt

Deploys CogAgent using vLLM for high-performance inference, offering improved throughput. Specify the model path, host, and port.

```bash
python app/vllm_openai_server.py \
    --model_path THUDM/cogagent-9b-20241220 \
    --host 0.0.0.0 \
    --port 8000
```

--------------------------------

### Multi-Card Fine-Tuning with DeepSpeed

Source: https://github.com/zai-org/cogagent/blob/main/finetune/README_en.md

Execute multi-card/multi-machine fine-tuning using DeepSpeed. Ensure DeepSpeed is installed and the model dataset is prepared.

```shell
OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=8  finetune.py  data/CogAgentData  THUDM/cogagent-9b-20241220  configs/sft.yaml
```

--------------------------------

### Start OpenAI-Compatible API Server

Source: https://context7.com/zai-org/cogagent/llms.txt

Deploys CogAgent as an OpenAI-compatible API server using FastAPI and Transformers. Specify the model path, host, and port.

```bash
python app/openai_demo.py \
    --model_path THUDM/cogagent-9b-20241220 \
    --host 0.0.0.0 \
    --port 8000
```

--------------------------------

### Verify Tkinter Installation

Source: https://github.com/zai-org/cogagent/blob/main/app/README_en.md

Verifies if the system's Python installation includes the tkinter library. This command should output version information if tkinter is correctly installed.

```shell
/opt/homebrew/bin/python3 -m tkinter
```

--------------------------------

### Install tkinter on macOS

Source: https://github.com/zai-org/cogagent/blob/main/app/README.md

Use Homebrew to install the tkinter library for Python on macOS. Ensure you have Homebrew installed first.

```shell
brew install python-tk
```

--------------------------------

### Load CogAgent Model with Transformers

Source: https://context7.com/zai-org/cogagent/llms.txt

Load the CogAgent model and tokenizer using HuggingFace Transformers for local inference. Requires at least 29GB VRAM for BF16 precision. The example demonstrates processing a screenshot and a query to generate an action.

```python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(
    "THUDM/cogagent-9b-20241220",
    trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    "THUDM/cogagent-9b-20241220",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto"
).eval()

# Load a screenshot
image = Image.open("screenshot.png").convert("RGB")

# Build the query prompt
task = "Click the search button"
platform_str = "(Platform: Mac)\n"  # Options: "Mac", "WIN", "Mobile"
format_str = "(Answer in Action-Operation-Sensitive format.)\n"
history_str = "\nHistory steps: "
query = f"Task: {task}{history_str}\n{platform_str}{format_str}"

# Tokenize and generate
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "image": image, "content": query}],
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
    return_dict=True
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_length=4096,
        do_sample=True,
        top_k=1
    )
    outputs = outputs[:, inputs["input_ids"].shape[1]:]
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(response)
# Output:
# Action: Click the search button in the top navigation bar.
# Grounded Operation: CLICK(box=[[352,102,786,139]], element_type='Button', element_info='Search')
# <<一般操作>>
```

--------------------------------

### Run Client Application

Source: https://github.com/zai-org/cogagent/blob/main/app/README.md

Launch the client application to connect to the running server. Ensure the server is accessible and provide the API key, base URL, client name, and port.

```shell
python client.py --api_key EMPTY --base_url http://127.0.0.1:7870/v1  --client_name 127.0.0.1 --client_port 7860 --model CogAgent
```

--------------------------------

### Launch Application

Source: https://github.com/zai-org/cogagent/blob/main/Action_space_en.md

Opens a specified application on the system. If a URL is also provided, the URL will take precedence.

```python
LAUNCH(app='Settings', url='None')
```

--------------------------------

### Run Server with Standard Python

Source: https://github.com/zai-org/cogagent/blob/main/app/README_en.md

Launches the CogAgent server using a standard Python script. Specify the model path, host, and port. This server mimics the OpenAI interface.

```shell
python openai_demo.py --model_path THUDM/cogagent-9b-20241220 --host 0.0.0.0 --port 7870
```

--------------------------------

### Launch URL

Source: https://github.com/zai-org/cogagent/blob/main/Action_space_en.md

Opens a given URL in the default web browser. If an application is specified, the URL parameter will be prioritized.

```python
LAUNCH(app='None', url='baidu.com')
```

--------------------------------

### Single-Card Fine-Tuning

Source: https://github.com/zai-org/cogagent/blob/main/finetune/README_en.md

Execute single-card fine-tuning. This command initiates the training process on a single device.

```shell
python finetune.py  data/CogAgentData/  THUDM/cogagent-9b-20241220  configs/lora.yaml
```

--------------------------------

### Run Gradio Web Demo

Source: https://context7.com/zai-org/cogagent/llms.txt

Launches an interactive web interface for testing CogAgent with image upload capabilities. Specify model directory, format key, platform, and output directory.

```bash
python inference/web_demo.py \
    --host 0.0.0.0 \
    --port 7860 \
    --model_dir THUDM/cogagent-9b-20241220 \
    --format_key status_action_op_sensitive \
    --platform "Mac" \
    --output_dir ./results
```

--------------------------------

### Run Server with vLLM

Source: https://github.com/zai-org/cogagent/blob/main/app/README_en.md

Launches the CogAgent server using vLLM for potentially improved performance. Similar to the standard Python method, specify model path, host, and port.

```shell
python vllm_openai_demo.py --model_path THUDM/cogagent-9b-20241220 --host 0.0.0.0 --port 7870
```

--------------------------------

### Enable Huawei Ascend NPU Support

Source: https://github.com/zai-org/cogagent/blob/main/finetune/README_en.md

Import necessary libraries to enable fine-tuning on Huawei Ascend NPU devices. Uncomment these lines before running the fine-tuning program.

```python
import torch_npu
from torch_npu.contrib import transfer_to_npu
```

--------------------------------

### Run CogAgent CLI Demo

Source: https://context7.com/zai-org/cogagent/llms.txt

Executes the CogAgent command-line interface demo with specified model directory, platform, and output settings. Includes options for different output formats.

```bash
# Basic CLI demo
python inference/cli_demo.py \
    --model_dir THUDM/cogagent-9b-20241220 \
    --platform "Mac" \
    --max_length 4096 \
    --top_k 1 \
    --output_image_path ./results \
    --format_key status_action_op_sensitive

# Available format_key options:
# - action_op_sensitive (default)
# - status_plan_action_op
# - status_action_op_sensitive
# - status_action_op
```

--------------------------------

### SFT Fine-tuning with DeepSpeed

Source: https://context7.com/zai-org/cogagent/llms.txt

Command for full supervised fine-tuning using DeepSpeed ZeRO-3 across 8 GPUs. Requires approximately 60GB VRAM per GPU.

```bash
# SFT fine-tuning on 8 GPUs (requires ~60GB VRAM per GPU)
OMP_NUM_THREADS=1 torchrun \
    --standalone \
    --nnodes=1 \
    --nproc_per_node=8 \
    finetune/finetune.py \
    data/CogAgentData \
    THUDM/cogagent-9b-20241220 \
    finetune/configs/sft.yaml
```

--------------------------------

### Launching Application or URL (LAUNCH)

Source: https://github.com/zai-org/cogagent/blob/main/Action_space_en.md

API for launching applications or opening URLs.

```APIDOC
## Launching an Application or a URL (LAUNCH)

### Description
Directly opens a specified application or a URL in a web browser.

### Method
Not applicable (function call syntax)

### Endpoint
Not applicable (function call syntax)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
1. `LAUNCH(app='Settings', url='None')`
2. `LAUNCH(app='None', url='baidu.com')`

### Response
#### Success Response (200)
None (operation executed)

#### Response Example
None
```

--------------------------------

### Format CogAgent User Input Prompts

Source: https://context7.com/zai-org/cogagent/llms.txt

Builds a CogAgent-compatible prompt by formatting task descriptions with history steps, platform information, and output format specification. The `identify_os` function helps determine the current operating system.

```python
def identify_os():
    """Identify operating system for platform string."""
    import platform
    os_detail = platform.platform().lower()
    if "mac" in os_detail:
        return "Mac"
    elif "windows" in os_detail:
        return "WIN"
    else:
        return "Mobile"

def format_cogagent_prompt(
    task: str,
    history_steps: list = None,
    history_actions: list = None,
    platform: str = None,
    format_key: str = "action_op_sensitive"
) -> str:
    """
    Build a CogAgent-compatible prompt.

    Args:
        task: Task description
        history_steps: List of previous grounded operations
        history_actions: List of previous action descriptions
        platform: "Mac", "WIN", or "Mobile"
        format_key: Output format type
    """
    # Format options
    format_dict = {
        "action_op_sensitive": "(Answer in Action-Operation-Sensitive format.)",
        "status_plan_action_op": "(Answer in Status-Plan-Action-Operation format.)",
        "status_action_op_sensitive": "(Answer in Status-Action-Operation-Sensitive format.)",
        "status_action_op": "(Answer in Status-Action-Operation format.)",
        "action_op": "(Answer in Action-Operation format.)"
    }

    current_platform = platform or identify_os()
    platform_str = f"(Platform: {current_platform})\n"
    format_str = format_dict.get(format_key, format_dict["action_op_sensitive"])

    # Build history string
    history_str = "\nHistory steps: "
    if history_steps and history_actions:
        for idx, (step, action) in enumerate(zip(history_steps, history_actions)):
            history_str += f"\n{idx}. {step}\t{action}"

    return f"Task: {task}{history_str}\n{platform_str}{format_str}"

# Example usage
prompt = format_cogagent_prompt(
    task="Search for doors, click doors on sale",
    history_steps=[
        "CLICK(box=[[352,102,786,139]], element_info='Search')",
        "TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')"
    ],
    history_actions=[
        "Left click on the search box",
        "Type 'doors' in the search input"
    ],
    platform="WIN",
    format_key="action_op"
)
print(prompt)
# Output:

```

--------------------------------

### Constructing User Input Prompt in Python

Source: https://github.com/zai-org/cogagent/blob/main/README.md

Dynamically builds a user query string by concatenating task, platform, format, and history. Ensure correct casing for platform identification.

```python
current_platform = identify_os() # "Mac" or "WIN" or "Mobile". Pay attention to case sensitivity.
platform_str = f"(Platform: {current_platform})\n"
format_str = "(Answer in Action-Operation-Sensitive format.)\n" # You can use other format to replace "Action-Operation-Sensitive"

history_str = "\nHistory steps: "
for index, (grounded_op_func, action) in enumerate(zip(history_grounded_op_funcs, history_actions)):
   history_str += f"\n{index}. {grounded_op_func}\t{action}" # start from 0. 

query = f"Task: {task}{history_str}\n{platform_str}{format_str}" # Be careful about the \n
```

--------------------------------

### Resume Fine-Tuning from Checkpoint

Source: https://github.com/zai-org/cogagent/blob/main/finetune/README_en.md

Continue fine-tuning from a previously saved checkpoint. Add 'yes' to resume from the last checkpoint or specify a checkpoint number.

```shell
python finetune.py data/CogAgent/ THUDM/cogagent-9b-20241220 configs/lora.yaml yes
```

--------------------------------

### Constructing User Input Prompt in Python

Source: https://github.com/zai-org/cogagent/blob/main/README_zh.md

This Python code demonstrates how to dynamically construct a user input prompt by concatenating task, history, platform, and format strings. Ensure correct newline characters for proper formatting.

```python
current_platform = identify_os() # "Mac" or "WIN" or "Mobile"，注意大小写
platform_str = f"(Platform: {current_platform})\n"
format_str = "(Answer in Action-Operation-Sensitive format.)\n" # You can use other format to replace "Action-Operation-Sensitive"

history_str = "\nHistory steps: "
for index, (grounded_op_func, action) in enumerate(zip(history_grounded_op_funcs, history_actions)):
   history_str += f"\n{index}. {grounded_op_func}\t{action}" # start from 0. 

query = f"Task: {task}{history_str}\n{platform_str}{format_str}" # Be careful about the \n

```

--------------------------------

### Run CogAgent Client

Source: https://github.com/zai-org/cogagent/blob/main/app/README_en.md

Executes the CogAgent client script. Ensure the server is running and accessible. Set the API key to EMPTY and provide the correct base URL for the server.

```shell
python client.py --api_key EMPTY --base_url http://127.0.0.1:7870/v1  --client_name 127.0.0.1 --client_port 7860 --model cogagent1.5-9b
```

--------------------------------

### LoRA Fine-tuning on Single GPU

Source: https://context7.com/zai-org/cogagent/llms.txt

Use this command to perform LoRA fine-tuning on a single GPU. Ensure you have sufficient VRAM (~70GB).

```bash
python finetune/finetune.py \
    data/CogAgentData/ \
    THUDM/cogagent-9b-20241220 \
    finetune/configs/lora.yaml
```

```bash
python finetune/finetune.py \
    data/CogAgentData/ \
    THUDM/cogagent-9b-20241220 \
    finetune/configs/lora.yaml \
    yes
```

--------------------------------

### Create Caches Directory

Source: https://github.com/zai-org/cogagent/blob/main/app/README_en.md

Creates a directory named 'caches' to store images generated during model execution. This is a necessary step for the demo's operation.

```shell
mkdir caches
```

--------------------------------

### Keyboard Press (KEY_PRESS)

Source: https://github.com/zai-org/cogagent/blob/main/Action_space_en.md

API for simulating a keyboard key press and release.

```APIDOC
## Keyboard Press (KEY_PRESS)

### Description
Simulates pressing and releasing a given keyboard key.

### Method
Not applicable (function call syntax)

### Endpoint
Not applicable (function call syntax)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
`KEY_PRESS(key='F11')`

### Response
#### Success Response (200)
None (operation executed)

#### Response Example
None
```

--------------------------------

### LoRA Fine-tuning Configuration

Source: https://context7.com/zai-org/cogagent/llms.txt

Configuration file for LoRA fine-tuning. Adjust parameters like learning rate, batch size, and LoRA specific settings.

```yaml
# finetune/configs/lora.yaml
data_config:
  train_file: train.jsonl
  val_file: dev.jsonl
  test_file: dev.jsonl
  num_proc: 1

combine: True
freezeV: False  # Vision encoder not frozen for LoRA
max_input_length: 512
max_output_length: 512

training_args:
  output_dir: ./output
  max_steps: 3000
  learning_rate: 5e-4
  per_device_train_batch_size: 1
  dataloader_num_workers: 16
  remove_unused_columns: false
  save_strategy: steps
  save_steps: 500
  log_level: info
  logging_steps: 10
  eval_strategy: steps
  eval_steps: 500
  predict_with_generate: true
  generation_config:
    max_new_tokens: 512

peft_config:
  peft_type: LORA
  task_type: CAUSAL_LM
  r: 8
  lora_alpha: 32
  lora_dropout: 0.1
  target_modules: ["query_key_value"]
```

--------------------------------

### SFT Fine-tuning Configuration

Source: https://context7.com/zai-org/cogagent/llms.txt

Configuration file for SFT fine-tuning. Note that the vision encoder is frozen (`freezeV: True`) and DeepSpeed is enabled.

```yaml
# finetune/configs/sft.yaml
data_config:
  train_file: train.jsonl
  val_file: dev.jsonl
  test_file: dev.jsonl
  num_proc: 1

combine: True
freezeV: True  # Vision encoder frozen for SFT
max_input_length: 512
max_output_length: 512

training_args:
  output_dir: ./output
  max_steps: 3000
  learning_rate: 5e-5
  per_device_train_batch_size: 1
  dataloader_num_workers: 16
  remove_unused_columns: false
  save_strategy: steps
  save_steps: 500
  logging_steps: 10
  eval_strategy: steps
  eval_steps: 500
  predict_with_generate: true
  generation_config:
    max_new_tokens: 512
  deepspeed: configs/ds_zero_3.json
```

--------------------------------

### Multi-round Fine-Tuning Data Format (JSON)

Source: https://github.com/zai-org/cogagent/blob/main/finetune/README_en.md

This JSON structure represents a single data entry for multi-round fine-tuning. It includes user messages with history steps and image references, followed by assistant responses. Ensure each entry has an image and that user history steps encompass all prior operations.

```json
{
  "messages": [
    {
      "role": "user",
      "content": "Task: In the Desktop & Dock section of System Settings, turn on the option 'Displays have separate Spaces' in the Mission Control section\n(Platform: Mac)\n(Answer in Action-Operation-Sensitive format.)\nHistory steps: \n0. CLICK(box=[[588,946,616,985]], element_info='System Settings')\tClick the 'System Settings' menu at the bottom of the screen to open System Settings\n1. CLICK(box=[[227,561,297,585]], element_info='Desktop & Dock')\tClick the 'Desktop & Dock' option in the left menu to enter the Desktop & Dock settings interface. \n2. SCROLL_DOWN(box=[[367,39,691,929]], step_count=75, element_info='Scroll')\tIn the scroll area on the right side of the current screen, scroll down to view more options. \n",
      "image": "images/0000000000336.png"
    },
    {
      "role": "assistant",
      "content": "Action: Click the 'Displays have separate spaces' option in the 'Schedule Center' section to enable this feature. \nGrounded Operation:CLICK(box=[[655,842,671,857]], element_info='```Displays have separate spaces``` switch button')\n<<General Operation>>"
    }
  ]
}
```

```json
{
  "messages": [
    {
      "role": "user",
      "content": "Task: Open the system settings through the launch pad, navigate to the 'Network' section, and set the DNS manual IP address to 8.8.4.4.\n(Platform: Mac)\n(Answer in Action-Operation-Sensitive format.)\nHistory steps: \n0. CLICK(box=[[7,6,27,31]], element_info='Apple logo')\tClick the 'Apple' icon in the upper left corner of the screen to open the system settings.\n1. CLICK(box=[[3,77,161,99]], element_info='System settings…')\tClick the 'System settings…' option in the menu bar in the upper left corner of the screen to open the system settings interface.\n2. CLICK(box=[[7,478,125,512]], element_info='Network')\tClick the 'Network' option in the left menu to enter the network settings interface. \n3. CLICK(box=[[155,217,459,270]], element_info='Wi-Fi, Connected')\tClick the 'Wi-Fi' option in the upper middle of the screen to enter the Wi-Fi settings interface. \n4. CLICK(box=[[405,280,447,296]], element_info='Details...')\tClick the 'Details...' button next to the Wi-Fi name to view detailed information about the current Wi-Fi. \n5. CLICK(box=[[27,344,45,362]], element_info='DNS')\tClick the 'DNS' option in the left menu to enter the DNS settings interface. \n6. CLICK(box=[[166,308,442,399]], element_info='8.8.8.8')\tClick the DNS server address input box to enter a new DNS server address. \n7. TYPE(box=[[163,308,443,322]], text='8.8.4.4', element_info='ip address input box')\tEnter 8.8.4.4 in the DNS server>Address input box to set a manual IP address. \n8. CLICK(box=[[410,636,440,651]], element_info='OK')\tClick the 'OK' button in the lower right corner of the interface to confirm and save the changes to the DNS settings. \n",
      "image": "images/0000000000482.png"
    },
    {
      "role": "assistant",
      "content": "Action: Left click\nGrounded Operation:CLICK(box=[[150,911,180,975]], element_info='lower')\n<<END>>"
    }
  ]
}
```

--------------------------------

### Training Data Format

Source: https://context7.com/zai-org/cogagent/llms.txt

Prepare training data in JSONL format, including image paths and conversation messages.

```APIDOC
## Training Data Format

### Description
Training data should be prepared in JSONL format. Each entry should contain a list of messages, where each message includes a role ('user' or 'assistant'), content, and an optional image path.

### Format
JSONL

### Example
```json
{
  "messages": [
    {
      "role": "user",
      "content": "Task: Search for 'CogAgent'\nHistory steps: \n(Platform: Mac)\n(Answer in Action-Operation format.)",
      "image": null
    },
    {
      "role": "assistant",
      "content": "Action: Click on the search box.\nGrounded Operation: CLICK(box=[[352,102,786,139]], element_type='Text input', element_info='Search')",
      "image": "/path/to/screenshot.png"
    }
  ]
}
```
```

--------------------------------

### Action Space Reference

Source: https://context7.com/zai-org/cogagent/llms.txt

Reference for the supported operations and their parameters in CogAgent.

```APIDOC
## Action Space Reference

### Description
CogAgent supports a variety of operations for interacting with graphical user interfaces. Each operation has specific parameters, including bounding box coordinates, text content, and element information.

### Supported Operations

#### Mouse Operations
- **CLICK**(box=[[x1,y1,x2,y2]], element_type='...', element_info='...')
- **DOUBLE_CLICK**(box=[[x1,y1,x2,y2]], element_type='...', element_info='...')
- **RIGHT_CLICK**(box=[[x1,y1,x2,y2]], element_type='...', element_info='...')
- **HOVER**(box=[[x1,y1,x2,y2]], element_type='...', element_info='...')

#### Text Input
- **TYPE**(box=[[x1,y1,x2,y2]], text='content', element_type='...', element_info='...')

#### Scrolling
- **SCROLL_UP**(box=[[x1,y1,x2,y2]], step_count=5, element_type='...', element_info='...')
- **SCROLL_DOWN**(box=[[x1,y1,x2,y2]], step_count=5, element_type='...', element_info='...')

#### Keyboard
- **KEY_PRESS**(key='Return')  # Keys: Return, Space, Up, Down, Left, Right, F1-F12, etc.

#### Combination Keys (Gestures)
- **GESTURE**(actions=[KEY_DOWN(key='Lcontrol'), KEY_PRESS(key='A'), KEY_UP(key='Lcontrol')])

#### Application Launch
- **LAUNCH**(app='Settings', url='None')
- **LAUNCH**(app='None', url='https://example.com')

#### Text Extraction
- **QUOTE_TEXT**(box=[[x1,y1,x2,y2]], output='__CogName_var__', result='extracted text')

#### Task Completion
- **END**()

### Box Format
- **box**: [[x1,y1,x2,y2]] where values are 0-999 (multiply by width/height / 1000)
```

--------------------------------

### Execute GUI Operations with PyAutoGUI

Source: https://context7.com/zai-org/cogagent/llms.txt

Provides Python functions to execute GUI operations like clicking, typing, and scrolling on the desktop using PyAutoGUI. Includes functions for identifying the OS and converting normalized bounding box coordinates to screen coordinates.

```python
import pyautogui
import pyperclip
import time
import platform

pyautogui.FAILSAFE = True
pyautogui.PAUSE = 1

def identify_os():
    os_detail = platform.platform().lower()
    return "Mac" if "mac" in os_detail else "Win"

def convert_box_to_screen_coords(box: list) -> tuple:
    """Convert normalized box coordinates to screen center point."""
    width, height = pyautogui.size()
    x_min = int(box[0] / 1000 * width)
    y_min = int(box[1] / 1000 * height)
    x_max = int(box[2] / 1000 * width)
    y_max = int(box[3] / 1000 * height)
    center_x = (x_min + x_max) / 2
    center_y = (y_min + y_max) / 2
    return (center_x, center_y)

def execute_click(box: list):
    """Execute a click at the center of the bounding box."""
    coords = convert_box_to_screen_coords(box)
    pyautogui.click(coords)

def execute_double_click(box: list):
    """Execute a double-click at the center of the bounding box."""
    coords = convert_box_to_screen_coords(box)
    pyautogui.doubleClick(coords)

def execute_right_click(box: list):
    """Execute a right-click at the center of the bounding box."""
    coords = convert_box_to_screen_coords(box)
    pyautogui.rightClick(coords)

def execute_type(box: list, text: str):
    """Type text at the specified location using clipboard paste."""
    coords = convert_box_to_screen_coords(box)
    pyautogui.click(coords)
    pyperclip.copy(text)
    time.sleep(0.5)
    if identify_os() == "Mac":
        with pyautogui.hold("command"):
            pyautogui.press("v")
    else:
        with pyautogui.hold("ctrl"):
            pyautogui.press("v")
    pyautogui.press("Return")

def execute_scroll(box: list, direction: str, steps: int = 5):
    """Scroll at the specified location."""
    pyautogui.moveTo(coords)
    scroll_amount = -steps if direction == "down" else steps
    pyautogui.scroll(scroll_amount)

def execute_key_press(key: str):
    """Press a keyboard key."""
    pyautogui.press(key)

def execute_hover(box: list):
    """Move mouse to hover over location."""
    coords = convert_box_to_screen_coords(box)
    pyautogui.moveTo(coords)

# Example: Execute a parsed operation
operation = {
    "operation": "CLICK",
    "box": [352, 102, 786, 139],
    "element_info": "Search"
}

if operation["operation"] == "CLICK":
    execute_click(operation["box"])
elif operation["operation"] == "TYPE":
    execute_type(operation["box"], operation.get("text", ""))
elif operation["operation"] == "SCROLL_DOWN":
    execute_scroll(operation["box"], "down", operation.get("step_count", 5))
```

--------------------------------

### SFT Fine-tuning with DeepSpeed

Source: https://context7.com/zai-org/cogagent/llms.txt

Perform full supervised fine-tuning with a frozen vision encoder using DeepSpeed ZeRO-3 across 8 GPUs. Requires approximately 60GB VRAM per GPU.

```APIDOC
## SFT Fine-tuning with DeepSpeed

### Description
This command performs full supervised fine-tuning (SFT) with a frozen vision encoder using DeepSpeed ZeRO-3. It is designed to run on 8 GPUs and requires approximately 60GB of VRAM per GPU.

### Method
Command Line

### Endpoint
N/A

### Parameters
#### Command Line Arguments
- **data_path** (string) - Required - Path to the training data.
- **model_name_or_path** (string) - Required - Name or path of the base model.
- **config_path** (string) - Required - Path to the SFT configuration file.

### Request Example
```bash
OMP_NUM_THREADS=1 torchrun \
    --standalone \
    --nnodes=1 \
    --nproc_per_node=8 \
    finetune/finetune.py \
    data/CogAgentData \
    THUDM/cogagent-9b-20241220 \
    finetune/configs/sft.yaml
```

### Configuration File (`finetune/configs/sft.yaml`)
```yaml
data_config:
  train_file: train.jsonl
  val_file: dev.jsonl
  test_file: dev.jsonl
  num_proc: 1

combine: True
freezeV: True  # Vision encoder frozen for SFT
max_input_length: 512
max_output_length: 512

training_args:
  output_dir: ./output
  max_steps: 3000
  learning_rate: 5e-5
  per_device_train_batch_size: 1
  dataloader_num_workers: 16
  remove_unused_columns: false
  save_strategy: steps
  save_steps: 500
  logging_steps: 10
  eval_strategy: steps
  eval_steps: 500
  predict_with_generate: true
  generation_config:
    max_new_tokens: 512
  deepspeed: configs/ds_zero_3.json
```
```

--------------------------------

### Running CogAgent with INT4/INT8 Precision

Source: https://github.com/zai-org/cogagent/blob/main/README_zh.md

This snippet shows how to enable INT4 or INT8 precision for inference. INT4 requires approximately 8GB of VRAM, and INT8 requires approximately 15GB. Using INT4 is not recommended due to significant performance degradation. This functionality is intended for NVIDIA devices.

```python
# To use INT4 or INT8 inference, uncomment the following lines in inference/cli_demo.py:
# python inference/cli_demo.py --model_path "path/to/your/model" --precision "int4" # or "int8"
```

--------------------------------

### Gestures (Combination Keys)

Source: https://github.com/zai-org/cogagent/blob/main/Action_space_en.md

API for performing complex gestures using key combinations.

```APIDOC
## Gestures (Combination Keys)

### Description
Performs gestures using combinations of key presses, such as Ctrl+F.

### Method
Not applicable (function call syntax)

### Endpoint
Not applicable (function call syntax)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
`GESTURE(actions=[KEY_DOWN(key='Lcontrol'), KEY_PRESS(key='A'), KEY_UP(key='Lcontrol')])`

### Response
#### Success Response (200)
None (operation executed)

#### Response Example
None
```

--------------------------------

### Calling a Large Language Model (LLM) and Using Variables

Source: https://github.com/zai-org/cogagent/blob/main/Action_space_en.md

This section describes how to call a large language model (LLM) to compute results, including parameters for prompts, output variables, and handling potentially long results.

```APIDOC
## LLM Call

### Description
Organize prompts and call a large language model to compute results.

### Method
LLM

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **prompt** (string) - Required - The prompt used to call the large language model, where the variable names used will be replaced with actual values.
- **output** (string) - Required - The variable name indicating where the result of the large language model call is stored. The format is `__CogName_xxx__`.
- **result** (string) - Optional - Represents the result of the large language model call. If the result is too long, the value of `result` will contain ellipses, or this parameter may not be present at all. In such cases, the CogAgent client-side application needs to call the large language model service to obtain the result.

### Request Example
```json
{
  "prompt": "Summarize the following content: __CogName_TechnicalReport__",
  "output": "__CogName_TechnicalReportSummary__"
}
```

### Response
#### Success Response (200)
- **output** (string) - The variable name where the result is stored.
- **result** (string) - The computed result from the LLM. May be truncated with ellipses if too long.

#### Response Example
```json
{
  "output": "__CogName_TechnicalReportSummary__",
  "result": "This is a summary of the technical report..."
}
```
```

--------------------------------

### SFT and LoRA Fine-tuning Requirements

Source: https://github.com/zai-org/cogagent/blob/main/README_zh.md

Details on fine-tuning CogAgent using SFT and LoRA. SFT requires freezing the Vision Encoder and uses 8xA100 GPUs with 60GB VRAM per GPU. LoRA does not freeze the Vision Encoder and uses 1xA100 GPU with 70GB VRAM. Both methods process 1600 tokens for the image and a total of 2048 tokens.

```python
# SFT fine-tuning: Freeze Vision Encoder, Batch Size = 1, 8 * A100 GPUs, 60GB VRAM/GPU
# LoRA fine-tuning: Unfreeze Vision Encoder, Batch Size = 1, 1 * A100 GPU, 70GB VRAM/GPU
# Input tokens (including image): 1600 tokens, Total tokens: 2048
```

--------------------------------

### Run Local Inference

Source: https://github.com/zai-org/cogagent/blob/main/README.md

Execute the command-line interface for local inference with CogAgent. Specify model directory, platform, max length, top k, output path, and response format.

```shell
python inference/cli_demo.py --model_dir THUDM/cogagent-9b-20241220 --platform "Mac" --max_length 4096 --top_k 1 --output_image_path ./results --format_key status_action_op_sensitive
```

--------------------------------

### Gesture Operation with Combination Keys

Source: https://github.com/zai-org/cogagent/blob/main/Action_space_en.md

Executes a sequence of key down, key press, and key up actions to simulate complex gestures like Ctrl+A. The 'actions' parameter takes a list of these key events.

```python
GESTURE(actions=[KEY_DOWN(key='Lcontrol'), KEY_PRESS(key='A'), KEY_UP(key='Lcontrol')])
```

--------------------------------

### CogAgent Action Space Reference

Source: https://context7.com/zai-org/cogagent/llms.txt

Reference for supported CogAgent operations, including mouse, text input, scrolling, keyboard, gestures, application launch, text extraction, and task completion. Box coordinates are normalized to 0-999.

```python
# Mouse Operations
CLICK(box=[[x1,y1,x2,y2]], element_type='...', element_info='...')
DOUBLE_CLICK(box=[[x1,y1,x2,y2]], element_type='...', element_info='...')
RIGHT_CLICK(box=[[x1,y1,x2,y2]], element_type='...', element_info='...')
HOVER(box=[[x1,y1,x2,y2]], element_type='...', element_info='...')

# Text Input
TYPE(box=[[x1,y1,x2,y2]], text='content', element_type='...', element_info='...')

# Scrolling (step_count: number of scroll wheel notches)
SCROLL_UP(box=[[x1,y1,x2,y2]], step_count=5, element_type='...', element_info='...')
SCROLL_DOWN(box=[[x1,y1,x2,y2]], step_count=5, element_type='...', element_info='...')

# Keyboard
KEY_PRESS(key='Return')  # Keys: Return, Space, Up, Down, Left, Right, F1-F12, etc.

# Combination Keys (Gestures)
GESTURE(actions=[KEY_DOWN(key='Lcontrol'), KEY_PRESS(key='A'), KEY_UP(key='Lcontrol')])

# Application Launch
LAUNCH(app='Settings', url='None')
LAUNCH(app='None', url='https://example.com')

# Text Extraction
QUOTE_TEXT(box=[[x1,y1,x2,y2]], output='__CogName_var__', result='extracted text')

# Task Completion
END()

# Box format: [[x1,y1,x2,y2]] where values are 0-999 (multiply by width/height / 1000)
```

--------------------------------

### Verify Virtual Environment tkinter

Source: https://github.com/zai-org/cogagent/blob/main/app/README.md

Confirm that tkinter is accessible within the newly created virtual environment. This ensures the virtual environment is set up correctly for GUI applications.

```shell
/Users/zr/Code/CogAgent/venv/bin/python3 -m tkinter
```

--------------------------------

### LoRA Fine-tuning

Source: https://context7.com/zai-org/cogagent/llms.txt

Fine-tune CogAgent using LoRA on a single GPU. Requires approximately 70GB of VRAM.

```APIDOC
## LoRA Fine-tuning on single GPU

### Description
This command initiates LoRA fine-tuning for CogAgent on a single GPU. It requires approximately 70GB of VRAM.

### Method
Command Line

### Endpoint
N/A

### Parameters
#### Command Line Arguments
- **data_path** (string) - Required - Path to the training data.
- **model_name_or_path** (string) - Required - Name or path of the base model.
- **config_path** (string) - Required - Path to the LoRA configuration file.
- **resume_from_checkpoint** (boolean) - Optional - If 'yes', resumes training from the latest checkpoint.

### Request Example
```bash
python finetune/finetune.py \
    data/CogAgentData/ \
    THUDM/cogagent-9b-20241220 \
    finetune/configs/lora.yaml
```

### Request Example (Resume)
```bash
python finetune/finetune.py \
    data/CogAgentData/ \
    THUDM/cogagent-9b-20241220 \
    finetune/configs/lora.yaml \
    yes
```

### Configuration File (`finetune/configs/lora.yaml`)
```yaml
data_config:
  train_file: train.jsonl
  val_file: dev.jsonl
  test_file: dev.jsonl
  num_proc: 1

combine: True
freezeV: False  # Vision encoder not frozen for LoRA
max_input_length: 512
max_output_length: 512

training_args:
  output_dir: ./output
  max_steps: 3000
  learning_rate: 5e-4
  per_device_train_batch_size: 1
  dataloader_num_workers: 16
  remove_unused_columns: false
  save_strategy: steps
  save_steps: 500
  log_level: info
  logging_steps: 10
  eval_strategy: steps
  eval_steps: 500
  predict_with_generate: true
  generation_config:
    max_new_tokens: 512

peft_config:
  peft_type: LORA
  task_type: CAUSAL_LM
  r: 8
  lora_alpha: 32
  lora_dropout: 0.1
  target_modules: ["query_key_value"]
```
```

--------------------------------

### Quoting Clipboard Content (QUOTE_CLIPBOARD)

Source: https://github.com/zai-org/cogagent/blob/main/Action_space_en.md

This operation allows the model to quickly obtain and use the content of the clipboard, storing it in a variable for subsequent steps.

```APIDOC
## QUOTE_CLIPBOARD

### Description
Store the content of the clipboard in a variable for use in subsequent steps.

### Method
QUOTE_CLIPBOARD

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **output** (string) - Required - The variable name to store the clipboard content.
- **result** (string) - Optional - The content of the clipboard.

### Request Example
```json
{
  "output": "__CogName_QuickSortCode__"
}
```

### Response
#### Success Response (200)
- **output** (string) - The variable name where the clipboard content is stored.
- **result** (string) - The content copied from the clipboard.

#### Response Example
```json
{
  "output": "__CogName_QuickSortCode__",
  "result": "def quick_sort(arr):\n\tif len(arr) <= 1:\n\t\treturn arr\n\t..."
}
```
```

--------------------------------

### Prompt Construction for Continued Answering

Source: https://github.com/zai-org/cogagent/blob/main/README_zh.md

To enable the 'continue answering' functionality, append '[Continue]\n' to the task field. This signals the model to resume its previous response.

```python
query = f'{task}[Continue]\n{history}{platform}{format}'
```

--------------------------------

### Create Virtual Environment

Source: https://github.com/zai-org/cogagent/blob/main/app/README_en.md

Creates a Python virtual environment using the venv module. Avoid using conda or virtualenv as they may interfere with tkinter. Replace the path with your desired location.

```shell
/opt/homebrew/bin/python3 -m venv --copies /Users/zr/Code/CogAgent/venv
```
Tokens:
Raw