Open-AutoGLM

https://github.com/zai-org/open-autoglm
Admin
Open-AutoGLM is a multimodal phone agent framework that understands smartphone screens and...
Tokens:8,044
Snippets:63
Trust Score:7.8
License:Apache-2.0
Update:10 months ago
Show doc for...
Tokens:
Raw
### Install Development Dependencies

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

This bash command installs the necessary development dependencies for the project, including optional dependencies specified with `.[dev]`. This is a prerequisite for setting up the development environment and running tests.

```bash
pip install -e ".[dev]"
```

--------------------------------

### Verbose Mode Agent Output Example

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

Illustrates the detailed output generated when Agent's `verbose` mode is enabled. It shows the AI's thinking process and the specific actions being executed, such as launching an app or tapping an element. This is useful for debugging and understanding the agent's behavior step-by-step.

```text
==================================================
💭 Thinking Process:
--------------------------------------------------
Currently on the system home screen, need to launch the Xiaohongshu app first
--------------------------------------------------
🎯 Executing Action:
{
  "_metadata": "do",
  "action": "Launch",
  "app": "Xiaohongshu"
}
==================================================

... (continue to next step after executing action)

==================================================
💭 Thinking Process:
--------------------------------------------------
Xiaohongshu is open, now need to tap the search box
--------------------------------------------------
🎯 Executing Action:
{
  "_metadata": "do",
  "action": "Tap",
  "element": [500, 100]
}
==================================================

🎉 ================================================
✅ Task Complete: Successfully searched for food guides
==================================================
```

--------------------------------

### Start vLLM Model Service

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

Starts the vLLM inference server for the AutoGLM model. This command requires the model to be downloaded and the inference engine installed. It exposes an OpenAI-compatible API endpoint on port 8000.

```shell
python3 -m vllm.entrypoints.openai.api_server \
 --served-model-name autoglm-phone-9b \
 --allowed-local-media-path /   \
 --mm-encoder-tp-mode data \
 --mm_processor_cache_type shm \
 --mm_processor_kwargs "{\"max_pixels\":5000000}" \
 --max-model-len 25480  \
 --chat-template-content-format string \
 --limit-mm-per-prompt "{\"image\":10}" \
 --model zai-org/AutoGLM-Phone-9B \
 --port 8000
```

--------------------------------

### Manage ADB Connections for Device Interaction

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Provides examples of managing Android Debug Bridge (ADB) connections, including connecting to remote devices via WiFi, listing all connected devices, checking connection status, and disconnecting. This is essential for the PhoneAgent to communicate with the Android device.

```python
from phone_agent.adb import ADBConnection, list_devices

# Create connection manager
conn = ADBConnection()

# Connect to remote device via WiFi
success, message = conn.connect("192.168.1.100:5555")
if success:
    print(f"✓ {message}")
else:
    print(f"✗ Failed: {message}")

# List all connected devices
devices = list_devices()
for device in devices:
    print(f"Device: {device.device_id}")
    print(f"  Status: {device.status}")
    print(f"  Type: {device.connection_type.value}")
    print(f"  Model: {device.model}")

# Check if specific device is connected
if conn.is_connected("192.168.1.100:5555"):
    print("Device is connected")

# Disconnect from remote device
success, message = conn.disconnect("192.168.1.100:5555")
print(message)
```

--------------------------------

### Enable TCP/IP Mode and Connect via WiFi

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Enables TCP/IP mode on a USB-connected device for WiFi debugging. It retrieves the device's IP address and attempts to connect to it wirelessly. This allows for disconnecting the USB cable after initial setup.

```python
success, message = conn.enable_tcpip(port=5555)
if success:
    print(f"✓ {message}")

    # Get device IP address
    ip = conn.get_device_ip()
    if ip:
        print(f"Device IP: {ip}")
        print(f"Connect using: adb connect {ip}:5555")
        print(f"Or in Python: conn.connect('{ip}:5555')")

        # Now you can disconnect USB and connect via WiFi
        import time
        time.sleep(2)
        success, msg = conn.connect(f"{ip}:5555")
        print(f"WiFi connection: {msg}")
else:
    print(f"✗ Failed: {message}")
```

--------------------------------

### Configure Remote ADB Connection via Python API

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

Python code for managing ADB connections programmatically. It includes functions to connect, list devices, enable TCP/IP mode, get device IP, and disconnect.

```python
from phone_agent.adb import ADBConnection, list_devices

# Create connection manager
conn = ADBConnection()

# Connect to remote device
success, message = conn.connect("192.168.1.100:5555")
print(f"Connection status: {message}")

# List connected devices
devices = list_devices()
for device in devices:
    print(f"{device.device_id} - {device.connection_type.value}")

# Enable TCP/IP on USB device
success, message = conn.enable_tcpip(5555)
ip = conn.get_device_ip()
print(f"Device IP: {ip}")

# Disconnect
conn.disconnect("192.168.1.100:5555")
```

--------------------------------

### Instantiate PhoneAgent with Custom Callbacks

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

This Python snippet shows how to create an instance of the PhoneAgent, integrating the previously defined custom confirmation and takeover callbacks. This allows the agent to use user-defined logic for handling sensitive operations and manual takeovers.

```python
agent = PhoneAgent(
    confirmation_callback=my_confirmation,
    takeover_callback=my_takeover,
)
```

--------------------------------

### Configure Model Settings with ModelConfig

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

This Python snippet demonstrates how to initialize and configure the ModelConfig object. It includes setting the base URL, API key, model name, maximum tokens, temperature, and frequency penalty. Ensure the `phone_agent.model` module is available.

```python
from phone_agent.model import ModelConfig

config = ModelConfig(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",  # API key (if required)
    model_name="autoglm-phone-9b",  # Model name
    max_tokens=3000,  # Maximum output tokens
    temperature=0.1,  # Sampling temperature
    frequency_penalty=0.2,  # Frequency penalty
)
```

--------------------------------

### Configure Agent Settings with AgentConfig

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

This Python snippet shows how to set up the AgentConfig. It allows customization of the maximum steps per task, the device ID for ADB connection (or auto-detection), and verbosity for debugging information. The `phone_agent.agent` module is required.

```python
from phone_agent.agent import AgentConfig

config = AgentConfig(
    max_steps=100,  # Max steps per task
    device_id=None,  # ADB device ID (None for auto-detect)
    verbose=True,  # Print debug info (including thinking process and actions)
)
```

--------------------------------

### Execute Batch Tasks with PhoneAgent in Python

Source: https://context7.com/zai-org/open-autoglm/llms.txt

This Python code demonstrates how to initialize the PhoneAgent with custom configurations and execute a list of tasks sequentially. Each task is run, its result or error is recorded, and the agent's state is reset. Finally, a summary of the batch execution is printed.

```python
from open_autoglm.agent import PhoneAgent
from open_autoglm.agent_config import AgentConfig

# Create agent with custom configuration
agent = PhoneAgent(
    agent_config=AgentConfig(
        max_steps=50,
        verbose=False  # Disable verbose for batch processing
    )
)

# Define task list
tasks = [
    "Open Amap and check traffic conditions",
    "Open Dianping and search for nearby cafes",
    "Open bilibili and search for Python tutorials",
    "Open Meituan and browse restaurants",
    "Open Taobao and search for keyboards"
]

results = []
for i, task in enumerate(tasks, 1):
    print(f"\n[{i}/{len(tasks)}] Executing: {task}")
    try:
        result = agent.run(task)
        results.append({"task": task, "result": result, "success": True})
        print(f"✓ Completed: {result}")
    except Exception as e:
        results.append({"task": task, "error": str(e), "success": False})
        print(f"✗ Failed: {e}")
    finally:
        agent.reset()  # Reset state for next task

# Summary
print(f"\n{'='*60}")
print(f"Batch Execution Summary: {sum(r['success'] for r in results)}/{len(results)} succeeded")
for r in results:
    status = "✓" if r["success"] else "✗"
    print(f"{status} {r['task']}")
```

--------------------------------

### Initialize and Run PhoneAgent with AI Model Configuration

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Demonstrates how to initialize the PhoneAgent with custom AI model configurations, including API endpoint, model name, and generation parameters. It then executes a natural language command to automate a task on the Android device. This snippet requires the 'phone_agent' library and a running AI model server.

```python
from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig
from phone_agent.agent import AgentConfig

# Configure the AI model endpoint
model_config = ModelConfig(
    base_url="http://localhost:8000/v1",
    model_name="autoglm-phone-9b",
    max_tokens=3000,
    temperature=0.1,
    frequency_penalty=0.2
)

# Configure agent behavior
agent_config = AgentConfig(
    max_steps=100,
    device_id=None,  # Auto-detect device
    verbose=True     # Print thinking process and actions
)

# Create agent and execute task
agent = PhoneAgent(
    model_config=model_config,
    agent_config=agent_config
)

result = agent.run("Open Taobao and search for wireless headphones")
print(f"Result: {result}")

# Reset for new task
agent.reset()
result = agent.run("Open Meituan and search for nearby hotpot restaurants")
```

--------------------------------

### Execute PhoneAgent Operations Step-by-Step

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Illustrates how to execute agent operations one step at a time using the `step()` method. This is useful for debugging and manual control, allowing observation of each action and thinking process. The loop continues until the task is finished or a maximum step count is reached.

```python
from phone_agent import PhoneAgent

agent = PhoneAgent()

# First step requires task description
result = agent.step("Open WeChat and check messages")
print(f"Step {agent.step_count}: Action={result.action}")
print(f"Thinking: {result.thinking}")
print(f"Success: {result.success}, Finished: {result.finished}")

# Continue execution step by step
while not result.finished and agent.step_count < 10:
    result = agent.step()
    print(f"Step {agent.step_count}: {result.action}")

    if result.message:
        print(f"Message: {result.message}")

print(f"Final result: {result.message}")
```

--------------------------------

### Command Line Interface for Automation Tasks

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Provides a command-line interface (CLI) for running automation tasks. It supports various modes, including interactive mode, executing single tasks, specifying target devices, connecting to remote devices, enabling WiFi debugging, listing devices and apps, and controlling verbosity.

```bash
# Interactive mode
python main.py --base-url http://localhost:8000/v1 --model autoglm-phone-9b

# Execute single task
python main.py --base-url http://localhost:8000/v1 "Open Dianping and search for nearby coffee shops"

# Specify device in multi-device setup
python main.py --device-id emulator-5554 "Open WeChat"

# Connect to remote device
python main.py --connect 192.168.1.100:5555

# Enable WiFi debugging on USB device
python main.py --enable-tcpip 5555

# List connected devices
python main.py --list-devices

# List supported apps
python main.py --list-apps

# Quiet mode (suppress verbose output)
python main.py --quiet "Open Taobao"
```

--------------------------------

### Manage Devices via ADB Commands

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

ADB commands for managing connected devices, including listing, connecting, disconnecting, and executing tasks on specific devices through the AutoGLM agent.

```bash
# List all connected devices
adb devices

# Connect to remote device
adb connect 192.168.1.100:5555

# Disconnect specific device
adb disconnect 192.168.1.100:5555

# Execute task on specific device
python main.py --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" "Open Douyin and browse videos"
```

--------------------------------

### Enable WiFi Debugging via ADB

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Demonstrates how to enable TCP/IP debugging on an Android device connected via USB using ADB. This allows for wireless ADB connections, which is a prerequisite for remote device control over WiFi.

```python
from phone_agent.adb import ADBConnection

conn = ADBConnection()

# Note: The actual command to enable WiFi debugging is usually run via conn.shell() or similar,
# but this snippet only shows initialization of ADBConnection for this purpose.
# Example of enabling: 
# success, message = conn.shell('adb tcpip 5555')
# print(message)

```

--------------------------------

### Interact with AutoGLM via Command Line

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

Command-line interface for interacting with the AutoGLM model. Users can specify the model's base URL and name for interactive use, direct task execution, or listing supported applications.

```bash
# Interactive mode
python main.py --base-url http://localhost:8000/v1 --model "autoglm-phone-9b"

# Specify model endpoint
python main.py --base-url http://localhost:8000/v1 "Find the top-rated cinema nearby and navigate me to there by foot"

# List supported apps
python main.py --list-apps
```

--------------------------------

### Configure Remote ADB Debugging

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

Commands for establishing and verifying ADB connections over WiFi for remote debugging. This allows device control without a USB connection.

```bash
# Connect via WiFi, change to the IP address and port shown on your phone
adb connect 192.168.1.100:5555

# Verify connection
# Should show: 192.168.1.100:5555    device
adb devices
```

--------------------------------

### Environment Variables for Agent Configuration

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Allows configuring the PhoneAgent using environment variables, which is useful for deployment automation. Key variables include the model's base URL, model name, maximum steps, and device ID. These can be set in the shell or within a Python script.

```bash
# Set model endpoint
export PHONE_AGENT_BASE_URL="http://192.168.1.50:8000/v1"
export PHONE_AGENT_MODEL="autoglm-phone-9b"
export PHONE_AGENT_MAX_STEPS="100"
export PHONE_AGENT_DEVICE_ID="192.168.1.100:5555"

# Run with environment configuration
python main.py "Open Douyin and browse videos"

# Or in Python
import os
os.environ["PHONE_AGENT_BASE_URL"] = "http://localhost:8000/v1"
os.environ["PHONE_AGENT_MODEL"] = "autoglm-phone-9b"

from phone_agent import PhoneAgent
agent = PhoneAgent()  # Uses environment variables
result = agent.run("Open bilibili")
```

--------------------------------

### Direct Model Inference with OpenAI-compatible Models

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Utilizes the ModelClient to perform direct inference with OpenAI-compatible vision-language models. It involves configuring the model, capturing screenshots, building messages with system and user prompts (including image data), and then requesting a response from the model.

```python
from phone_agent.model import ModelClient, ModelConfig, MessageBuilder
from phone_agent.adb import get_screenshot

# Configure model
config = ModelConfig(
    base_url="http://localhost:8000/v1",
    model_name="autoglm-phone-9b",
    max_tokens=3000,
    temperature=0.0,
    top_p=0.85
)

# Create client
client = ModelClient(config)

# Capture screen and build messages
screenshot = get_screenshot()
messages = [
    MessageBuilder.create_system_message(
        "You are an AI assistant that controls Android phones. Analyze the screen and suggest actions."
    ),
    MessageBuilder.create_user_message(
        text="What app is currently open? What actions can I take?",
        image_base64=screenshot.base64_data
    )
]

# Get model response
response = client.request(messages)
print(f"Thinking: {response.thinking}")
print(f"Action: {response.action}")
print(f"Raw: {response.raw_content}")
```

--------------------------------

### Run Project Tests

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

This bash command executes the project's test suite using pytest. It's essential for verifying the functionality and stability of the code after making changes or setting up the development environment.

```bash
pytest tests/
```

--------------------------------

### Interact with AutoGLM via Python API

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

Python code for initializing and using the PhoneAgent with a configured model. It demonstrates setting up model configuration and executing tasks using the agent.

```python
from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig

# Configure model
model_config = ModelConfig(
    base_url="http://localhost:8000/v1",
    model_name="autoglm-phone-9b",
)

# Create Agent
agent = PhoneAgent(model_config=model_config)

# Execute task
result = agent.run("Open Taobao and search for wireless earphones")
print(result)
```

--------------------------------

### Handle Sensitive Operations with Custom Callbacks

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Shows how to register custom callback functions for handling sensitive operations that require user confirmation or manual intervention (e.g., logins, CAPTCHAs). The `confirmation_callback` returns a boolean, while the `takeover_handler` allows for manual completion of a step.

```python
from phone_agent import PhoneAgent

def confirmation_handler(message: str) -> bool:
    """Called for sensitive operations requiring user confirmation"""
    print(f"\n⚠️  Sensitive Operation: {message}")
    response = input("Confirm? (y/n): ")
    return response.lower() in ("y", "yes")

def takeover_handler(message: str) -> None:
    """Called when manual intervention is needed (login, captcha, etc)"""
    print(f"\n🔐 Manual Intervention Required: {message}")
    input("Complete the operation manually and press Enter to continue...")

agent = PhoneAgent(
    confirmation_callback=confirmation_handler,
    takeover_callback=takeover_handler
)

# This may trigger confirmation if adding to cart
result = agent.run("Open Taobao, search for wireless earphones and add to cart")
print(f"Result: {result}")
```

--------------------------------

### Multi-Device Task Execution with PhoneAgent

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Enables automation tasks to be executed on specific devices in a multi-device environment. It includes listing connected devices, creating PhoneAgent instances for individual devices using their IDs, and then running automation tasks on each agent concurrently or sequentially.

```python
from phone_agent import PhoneAgent
from phone_agent.agent import AgentConfig
from phone_agent.adb import list_devices

# List all connected devices
devices = list_devices()
print("Connected devices:")
for dev in devices:
    print(f"  - {dev.device_id} ({dev.connection_type.value})")

# Create agents for specific devices
device_1_agent = PhoneAgent(
    agent_config=AgentConfig(
        device_id="emulator-5554",
        verbose=True
    )
)

device_2_agent = PhoneAgent(
    agent_config=AgentConfig(
        device_id="192.168.1.100:5555",
        verbose=True
    )
)

# Execute tasks on different devices
result1 = device_1_agent.run("Open bilibili and search for Python tutorials")
result2 = device_2_agent.run("Open Meituan and order food")

print(f"Device 1 result: {result1}")
print(f"Device 2 result: {result2}")
```

--------------------------------

### Low-Level Device Control Functions

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Provides a set of low-level functions to programmatically control an Android device. These functions include launching applications, performing tap, double-tap, and long-press actions, executing swipe gestures, and navigating the system using back and home buttons.

```python
from phone_agent.adb import device

# Launch an app
success = device.launch_app("微信", device_id=None)
if success:
    print("WeChat launched")

# Tap at coordinates (x, y in pixels)
device.tap(500, 1000, delay=1.0)

# Double tap
device.double_tap(500, 1000, delay=1.0)

# Long press (default 3 seconds)
device.long_press(500, 1000, duration_ms=3000, delay=1.0)

# Swipe from (x1, y1) to (x2, y2)
device.swipe(
    start_x=500, start_y=1500,
    end_x=500, end_y=500,
    duration_ms=1000,
    delay=1.0
)

# Press back button
device.back(delay=1.0)

# Press home button
device.home(delay=1.0)

# Get current app name
current_app = device.get_current_app(device_id=None)
print(f"Current app: {current_app}")
```

--------------------------------

### Batch Task Execution with PhoneAgent

Source: https://context7.com/zai-org/open-autoglm/llms.txt

Enables the sequential execution of multiple automation tasks using the PhoneAgent. This feature ensures proper state management across a series of tasks, allowing for complex automation workflows to be defined and run.

```python
from phone_agent import PhoneAgent
from phone_agent.agent import AgentConfig


```

--------------------------------

### Define Custom Takeover Callback

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

This Python function implements a custom callback for handling manual takeovers. It informs the user that a manual intervention is required and waits for their input before resuming. This is typically used for operations like CAPTCHAs or logins that the agent cannot handle automatically.

```python
def my_takeover(message: str) -> None:
    """Manual takeover callback"""
    print(f"Please complete manually: {message}")
    input("Press Enter to continue after completion...")
```

--------------------------------

### Define Custom Confirmation Callback

Source: https://github.com/zai-org/open-autoglm/blob/main/README_en.md

This Python function defines a custom callback for handling sensitive operation confirmations. It prompts the user for input before proceeding with potentially sensitive actions. The function returns a boolean indicating whether the operation should be confirmed.

```python
def my_confirmation(message: str) -> bool:
    """Sensitive operation confirmation callback"""
    return input(f"Confirm execution of {message}? (y/n): ").lower() == "y"
```
Tokens:
Raw