### Install App
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Installs an application on the device from a URL or local path.
```APIDOC
## POST /api/{platform}/{serial}/command/installApp
### Description
Installs an application on the device from a URL or local path.
### Method
POST
### Endpoint
/api/{platform}/{serial}/command/installApp
### Parameters
#### Path Parameters
- **platform** (string) - Required - The platform of the device (e.g., android, ios).
- **serial** (string) - Required - The serial number of the device.
#### Request Body
- **url** (string) - Required - The URL or local path to the application file.
### Request Example
```json
{
"url": "https://example.com/app.apk"
}
```
### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the installation was successful.
- **id** (string) - The ID of the installation task, if applicable.
#### Response Example
```json
{
"success": true,
"id": null
}
```
```
--------------------------------
### Install Application on Device
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Installs an application from a provided URL or local path onto the target device. Requires a JSON payload containing the source URL.
```bash
curl -X POST http://localhost:20242/api/android/emulator-5554/command/installApp -H "Content-Type: application/json" -d '{"url": "https://example.com/app.apk"}'
```
--------------------------------
### Install UIAutodev Package
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Install the UIAutodev package using pip. Includes an optional flag for HarmonyOS support.
```bash
pip install uiautodev
pip install "uiautodev[harmony]"
```
--------------------------------
### GET /api/info
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Retrieves general information about the running UIAutodev server instance.
```APIDOC
## GET /api/info
### Description
Returns information about the running UIAutodev server including version, platform, and available drivers.
### Method
GET
### Endpoint
/api/info
### Response
#### Success Response (200)
- **version** (string) - Server version
- **platform** (string) - Host OS
- **drivers** (array) - List of supported device drivers
#### Response Example
{
"version": "0.5.0",
"platform": "Linux",
"drivers": ["android", "ios", "harmony"]
}
```
--------------------------------
### Launch UIAutodev Local Server
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Commands to start the UIAutodev server with various configurations such as custom ports, offline mode, or headless operation.
```bash
uiauto.dev
uiauto.dev server --port 8080 --host 0.0.0.0
uiautodev server --offline
uiauto.dev server --no-browser
uiauto.dev server --force
```
--------------------------------
### Backup Application
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Downloads the APK file of an installed application from the device to the local machine for backup.
```bash
curl "http://localhost:20242/api/android/emulator-5554/backupApp?packageName=com.example.app" --output app_backup.apk
```
--------------------------------
### Automate HarmonyOS Device Interactions
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Provides an example of managing HarmonyOS devices, including capturing screenshots and launching applications with specific ability parameters.
```python
harmony_provider = HarmonyProvider()
harmony_devices = harmony_provider.list_devices()
if harmony_devices:
harmony_driver = harmony_provider.get_device_driver(harmony_devices[0].serial)
harmony_screenshot = harmony_driver.screenshot()
harmony_screenshot.save("harmony_screen.png")
harmony_driver.app_launch("com.example.app", page_name="MainAbility")
```
--------------------------------
### GET /api/{platform}/list
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Lists all connected devices for a specific mobile platform.
```APIDOC
## GET /api/{platform}/list
### Description
Lists all connected devices for a specific platform (android, ios, or harmony).
### Method
GET
### Endpoint
/api/{platform}/list
### Parameters
#### Path Parameters
- **platform** (string) - Required - The target platform (android, ios, or harmony)
### Response
#### Success Response (200)
- **serial** (string) - Device serial identifier
- **model** (string) - Device model name
- **status** (string) - Current connection status
#### Response Example
[
{
"serial": "emulator-5554",
"model": "sdk_gphone64_x86_64",
"status": "device"
}
]
```
--------------------------------
### Initialize App Inspector Status Check (JavaScript)
Source: https://github.com/codeskyblue/uiautodev/blob/master/uiautodev/static/demo.html
This script checks if the App Inspector Chrome Extension is installed by verifying the presence of `chrome.runtime`. It updates a message element on the page to indicate the installation status. This code is intended to run in a browser environment.
```JavaScript
window.onload = function() { let message = "App Inspector is installed."; if (chrome.runtime === undefined) { message = "Please install the App Inspector Chrome Extension to use this feature."; } document.getElementById('message').innerHTML = `${message}`; }
```
--------------------------------
### Get Current Application Information
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Retrieves the currently active application details on an iOS or Android device. Returns the package name, activity, and process ID.
```bash
curl http://localhost:20242/api/ios/00008030-001234567890/command/currentApp
```
--------------------------------
### Get Current Foreground App
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Retrieves information about the application currently running in the foreground on the specified device.
```bash
curl http://localhost:20242/api/android/emulator-5554/command/currentApp
```
--------------------------------
### GET /api/{platform}/{serial}/hierarchy
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Dumps the UI hierarchy tree of the device's current screen.
```APIDOC
## GET /api/{platform}/{serial}/hierarchy
### Description
Dumps the UI hierarchy tree of the device's current screen, supporting both JSON and XML formats.
### Method
GET
### Endpoint
/api/{platform}/{serial}/hierarchy
### Parameters
#### Path Parameters
- **platform** (string) - Required - The target platform
- **serial** (string) - Required - The device serial
#### Query Parameters
- **format** (string) - Optional - Output format (json or xml)
### Response
#### Success Response (200)
- **name** (string) - Element class name
- **bounds** (array) - Element coordinates
- **children** (array) - Nested UI elements
```
--------------------------------
### Get Current Activity
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Fetches the currently focused activity on the Android device.
```bash
curl http://localhost:20242/api/android/emulator-5554/current_activity
```
--------------------------------
### Get Current Activity
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Returns the currently focused activity on the Android device.
```APIDOC
## GET /api/android/{serial}/current_activity
### Description
Returns the currently focused activity on the Android device.
### Method
GET
### Endpoint
/api/android/{serial}/current_activity
### Parameters
#### Path Parameters
- **serial** (string) - Required - The serial number of the Android device.
### Response
#### Success Response (200)
- **activity** (string) - The name of the current activity.
#### Response Example
```
com.android.settings/.Settings
```
```
--------------------------------
### Get Current App on iOS
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Retrieves information about the currently active application on an iOS device.
```APIDOC
## GET /api/ios/{serial}/command/currentApp
### Description
Retrieves information about the currently active application on an iOS device.
### Method
GET
### Endpoint
/api/ios/{serial}/command/currentApp
### Parameters
#### Path Parameters
- **serial** (string) - Required - The serial number of the iOS device.
### Response
#### Success Response (200)
- **package** (string) - The package name of the current app.
- **activity** (string) - The activity name of the current app.
- **pid** (integer) - The process ID of the current app.
#### Response Example
```json
{
"package": "com.android.settings",
"activity": ".Settings",
"pid": 12345
}
```
```
--------------------------------
### Python API Usage
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Utilize UIAutodev drivers directly within Python scripts for building custom automation workflows. This section provides examples of importing and using the provider and command type classes.
```APIDOC
## Python API Usage
Use UIAutodev drivers directly in Python scripts for custom automation workflows.
### Importing Providers and Command Types
```python
from uiautodev.provider import AndroidProvider, IOSProvider, HarmonyProvider
from uiautodev.command_types import TapRequest, FindElementRequest
# Example usage (conceptual):
# android_provider = AndroidProvider()
# tap_command = TapRequest(x=100, y=200)
# android_provider.execute_command(tap_command)
```
```
--------------------------------
### GET /api/{platform}/features
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Retrieves a list of supported features and endpoints for a specified platform (e.g., Android, iOS). This is useful for understanding the capabilities available for a particular device type.
```APIDOC
## GET /api/{platform}/features
### Description
Returns the list of supported features/endpoints for a specific platform.
### Method
GET
### Endpoint
/api/{platform}/features
### Parameters
#### Path Parameters
- **platform** (string) - Required - The target platform (e.g., 'android', 'ios').
### Request Example
```bash
curl http://localhost:20242/api/android/features
```
### Response
#### Success Response (200)
- **screenshot** (boolean) - Indicates if screenshot functionality is supported.
- **hierarchy** (boolean) - Indicates if UI hierarchy retrieval is supported.
- **shell** (boolean) - Indicates if shell command execution is supported.
- **backupApp** (boolean) - Indicates if app backup functionality is supported.
- **current_activity** (boolean) - Indicates if retrieving the current activity is supported.
#### Response Example
```json
{
"screenshot": true,
"hierarchy": true,
"shell": true,
"backupApp": true,
"current_activity": true
}
```
```
--------------------------------
### Send Message to App Inspector Extension (JavaScript)
Source: https://github.com/codeskyblue/uiautodev/blob/master/uiautodev/static/demo.html
This script sends a message to the App Inspector Chrome Extension with the ID 'fjbboaelofjaabjmlphndicacmapbalm' to request URL information. It handles the response, logging any errors or displaying the description from the extension. This code requires the App Inspector extension to be installed and accessible.
```JavaScript
document.getElementById('test').addEventListener('click', function() { const extensionId = "fjbboaelofjaabjmlphndicacmapbalm"; chrome.runtime.sendMessage(extensionId, {url: "/info"}, function(response) { console.log(response); if (response.error) { console.error(response.error); } else { document.getElementById('message').innerHTML = response.data.description; } }); });
```
--------------------------------
### Initialize Python API Drivers
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Demonstrates how to import and initialize the core provider classes for Android, iOS, and HarmonyOS to build custom automation scripts.
```python
from uiautodev.provider import AndroidProvider, IOSProvider, HarmonyProvider
from uiautodev.command_types import TapRequest, FindElementRequest
```
--------------------------------
### Retrieve Server Information
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Fetches metadata about the running server instance, including version, platform, and active drivers.
```bash
curl http://localhost:20242/api/info
```
```json
{
"version": "0.5.0",
"description": "client for https://uiauto.dev",
"platform": "Linux",
"code_language": "Python",
"cwd": "/home/user/project",
"drivers": ["android", "ios", "harmony"]
}
```
--------------------------------
### Retrieve Supported Platform Features
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Fetches a list of supported capabilities and endpoints for a specific mobile platform. This is useful for verifying if features like screenshotting or hierarchy dumping are available.
```bash
curl http://localhost:20242/api/android/features
```
--------------------------------
### Automate Android Device Interactions
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Demonstrates how to connect to an Android device, capture screenshots, dump UI hierarchy, and perform common actions like tapping, text input, and app management.
```python
android_provider = AndroidProvider()
devices = android_provider.list_devices()
if devices:
driver = android_provider.get_device_driver(devices[0].serial)
screenshot = driver.screenshot(0)
screenshot.save("android_screen.png")
xml_data, hierarchy = driver.dump_hierarchy()
driver.tap(500, 800)
driver.app_launch("com.android.settings")
driver.send_keys("Hello World")
```
--------------------------------
### Execute CLI Automation Commands
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Provides common CLI commands for device management, including listing devices, performing touch interactions, app lifecycle management, and system navigation.
```bash
uiauto.dev android list
uiauto.dev android tap x=500 y=800
uiauto.dev android tap x=0.5 y=0.5 isPercent=true
uiauto.dev android currentApp
uiauto.dev android appLaunch package=com.android.settings
uiauto.dev android appTerminate package=com.android.settings
uiauto.dev android home
uiauto.dev android back
uiauto.dev android wakeUp
uiauto.dev android getWindowSize
uiauto.dev android dump
uiauto.dev android installApp url=https://example.com/app.apk
uiauto.dev version
uiauto.dev self-update
uiauto.dev shutdown
```
--------------------------------
### Automate iOS Device Interactions
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Shows the basic workflow for connecting to an iOS device, capturing a screenshot, and performing basic UI interactions like tapping and returning to the home screen.
```python
ios_provider = IOSProvider()
ios_devices = ios_provider.list_devices()
if ios_devices:
ios_driver = ios_provider.get_device_driver(ios_devices[0].serial)
ios_screenshot = ios_driver.screenshot()
ios_screenshot.save("ios_screen.png")
ios_driver.tap(200, 400)
ios_driver.home()
```
--------------------------------
### List Connected Devices
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Lists all connected devices for a specific platform. Supported platforms are android, ios, and harmony.
```bash
curl http://localhost:20242/api/android/list
curl http://localhost:20242/api/ios/list
curl http://localhost:20242/api/harmony/list
```
--------------------------------
### CLI Commands
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Execute device commands directly from the command line for automation scripts and quick testing. This section covers various commands for device management, app interaction, and UI automation.
```APIDOC
## CLI Commands
Execute device commands directly from the command line for automation scripts and quick testing.
### List Connected Devices
```bash
uiauto.dev android list
uiauto.dev ios list
```
### Tap Actions
Tap at specific coordinates or using percentage-based coordinates.
```bash
# Tap at coordinates (pixels)
uiauto.dev android tap x=500 y=800
# Tap using percentage coordinates
uiauto.dev android tap x=0.5 y=0.5 isPercent=true
```
### Application Management
Commands for interacting with applications on the device.
```bash
# Get current app package name
uiauto.dev android currentApp
# Launch an application
uiauto.dev android appLaunch package=com.android.settings
# Terminate an application
uiauto.dev android appTerminate package=com.android.settings
# Install app from URL
uiauto.dev android installApp url=https://example.com/app.apk
```
### Device Control
Commands for basic device control.
```bash
# Press the home button
uiauto.dev android home
# Press the back button
uiauto.dev android back
# Wake up the device
uiauto.dev android wakeUp
```
### Information Retrieval
Commands to get information about the device and its UI.
```bash
# Get window size
uiauto.dev android getWindowSize
# Dump UI hierarchy
uiauto.dev android dump
```
### Utility Commands
Commands for version checking, updating, and server control.
```bash
# Print version
uiauto.dev version
# Update to latest version
uiauto.dev self-update
# Shutdown running server
uiauto.dev shutdown
```
```
--------------------------------
### Execute Device Commands
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Performs various system-level operations such as navigation, volume control, and app lifecycle management. This is a unified interface for common device interactions.
```bash
curl -X POST http://localhost:20242/api/android/emulator-5554/command/home
curl -X POST http://localhost:20242/api/android/emulator-5554/command/appLaunch -H "Content-Type: application/json" -d '{"package": "com.android.settings", "stop": false}'
```
--------------------------------
### Environment Variables
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Configure UIAutodev behavior through environment variables. These variables allow customization of the driver, enabling mock mode, and other settings.
```APIDOC
## Environment Variables
Configure UIAutodev behavior through environment variables.
### UIAUTODEV_USE_ADB_DRIVER
Use ADB driver instead of uiautomator2. This can be faster but offers fewer features.
```bash
export UIAUTODEV_USE_ADB_DRIVER=1
```
### UIAUTODEV_MOCK
Enable mock mode for testing without real devices. This is useful for development and testing purposes.
```bash
export UIAUTODEV_MOCK=1
```
```
--------------------------------
### Execute Device Command
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Executes various device commands including navigation, volume control, and app management through a unified command interface.
```APIDOC
## POST /api/{platform}/{serial}/command/{command}
### Description
Executes various device commands including navigation, volume control, and app management through a unified command interface.
### Method
POST
### Endpoint
/api/{platform}/{serial}/command/{command}
### Parameters
#### Path Parameters
- **platform** (string) - Required - The platform of the device (e.g., android, ios).
- **serial** (string) - Required - The serial number of the device.
- **command** (string) - Required - The command to execute (e.g., home, back, wakeUp, volumeUp, appLaunch, sendKeys).
#### Request Body
- **package** (string) - Optional - The package name for app-related commands.
- **stop** (boolean) - Optional - Whether to stop the app before launching (for appLaunch command).
- **text** (string) - Optional - The text to send for sendKeys command.
### Request Example
```json
{
"package": "com.android.settings",
"stop": false
}
```
### Response
#### Success Response (200)
- **width** (integer) - The width of the device screen (for getWindowSize command).
- **height** (integer) - The height of the device screen (for getWindowSize command).
#### Response Example
```json
{"width": 1080, "height": 2400}
```
```
--------------------------------
### Connect to HarmonyOS MJPEG WebSocket
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Establishes a WebSocket connection to stream HarmonyOS device screens using the MJPEG format. Each received binary frame represents a complete JPEG image.
```python
import asyncio
import websockets
async def connect_harmony():
uri = "ws://localhost:20242/ws/harmony/mjpeg/HARMONY-SERIAL"
async with websockets.connect(uri) as ws:
while True:
data = await ws.recv()
if isinstance(data, bytes):
with open("frame.jpg", "wb") as f:
f.write(data)
print(f"Received frame: {len(data)} bytes")
asyncio.run(connect_harmony())
```
--------------------------------
### Perform Tap Action
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Sends a tap command to the device. Supports both absolute pixel coordinates and percentage-based positioning.
```bash
curl -X POST http://localhost:20242/api/android/emulator-5554/command/tap \
-H "Content-Type: application/json" \
-d '{"x": 500, "y": 800, "isPercent": false}'
curl -X POST http://localhost:20242/api/android/emulator-5554/command/tap \
-H "Content-Type: application/json" \
-d '{"x": 0.5, "y": 0.5, "isPercent": true}'
```
--------------------------------
### Find UI Elements
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Searches for UI elements in the current screen hierarchy using locators like ID, text, or class name. Supports setting a timeout for the search operation.
```bash
curl -X POST http://localhost:20242/api/android/emulator-5554/command/findElements -H "Content-Type: application/json" -d '{"by": "id", "value": "com.example:id/button", "timeout": 10.0}'
```
--------------------------------
### Capture Device Screenshot
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Captures a screenshot from a specific device and saves it as a JPEG file. Requires the platform, serial number, and display ID.
```bash
curl http://localhost:20242/api/android/emulator-5554/screenshot/0 --output screenshot.jpg
curl http://localhost:20242/api/ios/00008030-001234567890/screenshot/0 --output ios_screenshot.jpg
curl http://localhost:20242/api/harmony/HDC-SERIAL/screenshot/0 --output harmony_screenshot.jpg
```
--------------------------------
### Execute Shell Command
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Executes a shell command directly on the Android device via ADB.
```APIDOC
## POST /api/android/{serial}/shell
### Description
Executes a shell command directly on the Android device via ADB.
### Method
POST
### Endpoint
/api/android/{serial}/shell
### Parameters
#### Path Parameters
- **serial** (string) - Required - The serial number of the Android device.
#### Request Body
- **command** (string) - Required - The shell command to execute.
### Request Example
```json
{
"command": "pm list packages -3"
}
```
### Response
#### Success Response (200)
- **output** (string) - The standard output of the command.
- **error** (string) - The standard error of the command, if any.
#### Response Example
```json
{
"output": "package:com.example.app1\npackage:com.example.app2",
"error": null
}
```
```
--------------------------------
### Click UI Element
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Locates a specific UI element and performs a click action on its center point. Includes automatic retry logic until the specified timeout is reached.
```bash
curl -X POST http://localhost:20242/api/android/emulator-5554/command/clickElement -H "Content-Type: application/json" -d '{"by": "text", "value": "Submit", "timeout": 10.0}'
```
--------------------------------
### Dump UI Hierarchy
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Retrieves the UI hierarchy tree of the current screen. Supports JSON and XML formats for parsing element properties.
```bash
curl http://localhost:20242/api/android/emulator-5554/hierarchy
curl "http://localhost:20242/api/android/emulator-5554/hierarchy?format=xml"
```
--------------------------------
### POST /api/{platform}/{serial}/command/tap
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Performs a tap action on the device at specified coordinates.
```APIDOC
## POST /api/{platform}/{serial}/command/tap
### Description
Performs a tap action on the device at specified coordinates, supporting both absolute pixel coordinates and percentage-based positioning.
### Method
POST
### Endpoint
/api/{platform}/{serial}/command/tap
### Request Body
- **x** (number) - Required - X coordinate
- **y** (number) - Required - Y coordinate
- **isPercent** (boolean) - Required - Whether coordinates are percentages
### Request Example
{
"x": 500,
"y": 800,
"isPercent": false
}
### Response
#### Success Response (200)
- **status** (string) - Operation result
```
--------------------------------
### WebSocket /ws/harmony/mjpeg/{serial}
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Establishes a WebSocket connection for streaming HarmonyOS device screens in MJPEG format. This is useful for real-time monitoring of the device's display.
```APIDOC
## WebSocket /ws/harmony/mjpeg/{serial}
### Description
Establishes a WebSocket connection for HarmonyOS device screen streaming using MJPEG format.
### Method
WebSocket
### Endpoint
/ws/harmony/mjpeg/{serial}
### Parameters
#### Path Parameters
- **serial** (string) - Required - The serial number of the HarmonyOS device.
### Request Example
```python
import asyncio
import websockets
async def connect_harmony():
uri = "ws://localhost:20242/ws/harmony/mjpeg/HARMONY-SERIAL"
async with websockets.connect(uri) as ws:
while True:
# Receive MJPEG frames
data = await ws.recv()
if isinstance(data, bytes):
# Each frame is a complete JPEG image
with open("frame.jpg", "wb") as f:
f.write(data)
print(f"Received frame: {len(data)} bytes")
asyncio.run(connect_harmony())
```
### Response
#### Success Response (101) - Switching Protocols
- The connection is upgraded to WebSocket.
- MJPEG frames (JPEG images) are sent as binary data.
#### Response Example
(Binary MJPEG frame data)
```
--------------------------------
### Configure Environment Variables
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Sets environment variables to modify the behavior of the UIAutodev server, such as switching drivers or enabling mock modes for testing.
```bash
export UIAUTODEV_USE_ADB_DRIVER=1
export UIAUTODEV_MOCK=1
```
--------------------------------
### Backup App
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Downloads an APK file from the device for backup purposes.
```APIDOC
## GET /api/{platform}/{serial}/backupApp
### Description
Downloads an APK file from the device for backup purposes.
### Method
GET
### Endpoint
/api/{platform}/{serial}/backupApp
### Parameters
#### Path Parameters
- **platform** (string) - Required - The platform of the device (e.g., android, ios).
- **serial** (string) - Required - The serial number of the device.
#### Query Parameters
- **packageName** (string) - Required - The package name of the app to back up.
### Response
#### Success Response (200)
- The response will be the APK file content.
### Request Example
```bash
curl "http://localhost:20242/api/android/emulator-5554/backupApp?packageName=com.example.app" \
--output app_backup.apk
```
```
--------------------------------
### WebSocket Video Stream Rendering
Source: https://github.com/codeskyblue/uiautodev/blob/master/examples/harmony-video.html
This snippet initializes a WebSocket connection to receive binary image data, converts it into a Blob, and renders it onto an HTML5 canvas while maintaining aspect ratio.
```javascript
const ws = new WebSocket('ws://localhost:8899');
ws.binaryType = 'arraybuffer';
ws.onmessage = function(event) {
const arrayBuffer = event.data;
const blob = new Blob([arrayBuffer], {type: 'image/jpeg'});
const url = URL.createObjectURL(blob);
const img = new Image();
img.onload = function() {
const containerWidth = canvasContainer.offsetWidth;
const containerHeight = canvasContainer.offsetHeight;
const aspectRatio = img.width / img.height;
if (containerWidth / containerHeight > aspectRatio) {
canvas.height = containerHeight;
canvas.width = Math.floor(containerHeight * aspectRatio);
} else {
canvas.width = containerWidth;
canvas.height = Math.floor(containerWidth / aspectRatio);
}
context.clearRect(0, 0, canvas.width, canvas.height);
context.drawImage(img, 0, 0, canvas.width, canvas.height);
URL.revokeObjectURL(url);
};
img.src = url;
};
```
--------------------------------
### Execute Shell Command
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Runs arbitrary shell commands directly on the Android device via ADB. Returns the standard output and error stream.
```bash
curl -X POST http://localhost:20242/api/android/emulator-5554/shell -H "Content-Type: application/json" -d '{"command": "pm list packages -3"}'
```
--------------------------------
### Find Elements
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Searches for UI elements in the current hierarchy using various locator strategies.
```APIDOC
## POST /api/{platform}/{serial}/command/findElements
### Description
Searches for UI elements in the current hierarchy using various locator strategies including ID, text, class name, and XPath.
### Method
POST
### Endpoint
/api/{platform}/{serial}/command/findElements
### Parameters
#### Path Parameters
- **platform** (string) - Required - The platform of the device (e.g., android, ios).
- **serial** (string) - Required - The serial number of the device.
#### Request Body
- **by** (string) - Required - The locator strategy (e.g., id, text, className, xpath).
- **value** (string) - Required - The value to search for.
- **timeout** (float) - Optional - The timeout in seconds for the search.
### Request Example
```json
{
"by": "id",
"value": "com.example:id/button",
"timeout": 10.0
}
```
### Response
#### Success Response (200)
- **count** (integer) - The number of elements found.
- **value** (array) - An array of found UI elements.
- **key** (string) - Unique identifier for the element.
- **name** (string) - The name or class of the element.
- **bounds** (array) - The bounding box of the element [x1, y1, x2, y2].
- **properties** (object) - Key-value pairs of element properties.
- **children** (array) - Child elements of the current element.
#### Response Example
```json
{
"count": 1,
"value": [
{
"key": "0-1-2",
"name": "android.widget.Button",
"bounds": [0.1, 0.3, 0.9, 0.35],
"properties": {
"resource-id": "com.example:id/button",
"text": "Click Me",
"clickable": "true"
},
"children": []
}
]
}
```
```
--------------------------------
### Canvas Coordinate Mapping and Event Handling
Source: https://github.com/codeskyblue/uiautodev/blob/master/examples/harmony-video.html
This snippet captures mouse events on the canvas, calculates the scaled coordinates relative to the target device screen size, and sends the action to the backend via WebSocket.
```javascript
function calculateCoordinates(event) {
const rect = canvas.getBoundingClientRect();
const mouseXCanvas = event.clientX - rect.left;
const mouseYCanvas = event.clientY - rect.top;
const scaleX = phoneScreenWidth / canvas.width;
const scaleY = phoneScreenHeight / canvas.height;
return {
mouseX: Math.floor(mouseXCanvas * scaleX),
mouseY: Math.floor(mouseYCanvas * scaleY)
};
}
canvas.addEventListener('mouseup', function(event) {
upCoordinates = calculateCoordinates(event);
if (downCoordinates.mouseX === upCoordinates.mouseX && downCoordinates.mouseY === upCoordinates.mouseY) {
sendMouseEvent('normal', downCoordinates.mouseX, downCoordinates.mouseY);
} else {
sendMouseMoveEvent('move', downCoordinates.mouseX, downCoordinates.mouseY, upCoordinates.mouseX, upCoordinates.mouseY);
}
});
```
--------------------------------
### WebSocket /ws/android/scrcpy/{serial}
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Establishes a WebSocket connection for real-time screen mirroring and touch control using the scrcpy protocol for Android devices. Allows sending touch events, key events, and text, while receiving video streams.
```APIDOC
## WebSocket /ws/android/scrcpy/{serial}
### Description
Establishes a WebSocket connection for real-time screen mirroring and touch control using scrcpy protocol.
### Method
WebSocket
### Endpoint
/ws/android/scrcpy/{serial}
### Parameters
#### Path Parameters
- **serial** (string) - Required - The serial number of the Android device.
#### Query Parameters
- **version** (string) - Optional - The scrcpy protocol version.
### Request Example
```python
import asyncio
import websockets
import json
async def connect_scrcpy():
uri = "ws://localhost:20242/ws/android/scrcpy/emulator-5554?version=2.7"
async with websockets.connect(uri) as ws:
# Receive video stream (binary H264 data)
async def receive_video():
while True:
data = await ws.recv()
if isinstance(data, bytes):
# Process H264 video frame
print(f"Received {len(data)} bytes of video data")
# Send touch events
async def send_touch():
# Touch down at center
await ws.send(json.dumps({
"type": "touchDown",
"xP": 0.5, # 50% from left
"yP": 0.5 # 50% from top
}))
await asyncio.sleep(0.1)
# Touch up
await ws.send(json.dumps({
"type": "touchUp",
"xP": 0.5,
"yP": 0.5
}))
# Send key event (HOME = 3)
await ws.send(json.dumps({
"type": "keyEvent",
"data": {"eventNumber": 3}
}))
# Send text input
await ws.send(json.dumps({
"type": "text",
"detail": "Hello from WebSocket"
}))
# Ping to keep connection alive
await ws.send(json.dumps({"type": "ping"}))
await asyncio.gather(receive_video(), send_touch())
asyncio.run(connect_scrcpy())
```
### Response
#### Success Response (101) - Switching Protocols
- The connection is upgraded to WebSocket.
- Data frames will be sent and received according to the scrcpy protocol.
#### Response Example
(No direct response body, but data frames are exchanged)
```json
{
"type": "touchDown",
"xP": 0.5,
"yP": 0.5
}
```
```
--------------------------------
### Connect to Android Scrcpy WebSocket
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Establishes a WebSocket connection for real-time screen mirroring and remote control. It handles binary H264 video streams and sends touch, key, and text input events.
```python
import asyncio
import websockets
import json
async def connect_scrcpy():
uri = "ws://localhost:20242/ws/android/scrcpy/emulator-5554?version=2.7"
async with websockets.connect(uri) as ws:
async def receive_video():
while True:
data = await ws.recv()
if isinstance(data, bytes):
print(f"Received {len(data)} bytes of video data")
async def send_touch():
await ws.send(json.dumps({"type": "touchDown", "xP": 0.5, "yP": 0.5}))
await asyncio.sleep(0.1)
await ws.send(json.dumps({"type": "touchUp", "xP": 0.5, "yP": 0.5}))
await ws.send(json.dumps({"type": "keyEvent", "data": {"eventNumber": 3}}))
await ws.send(json.dumps({"type": "text", "detail": "Hello from WebSocket"}))
await ws.send(json.dumps({"type": "ping"}))
await asyncio.gather(receive_video(), send_touch())
asyncio.run(connect_scrcpy())
```
--------------------------------
### Click Element
Source: https://context7.com/codeskyblue/uiautodev/llms.txt
Finds an element using the specified locator and clicks on its center point.
```APIDOC
## POST /api/{platform}/{serial}/command/clickElement
### Description
Finds an element using the specified locator and clicks on its center point, with automatic retry until timeout.
### Method
POST
### Endpoint
/api/{platform}/{serial}/command/clickElement
### Parameters
#### Path Parameters
- **platform** (string) - Required - The platform of the device (e.g., android, ios).
- **serial** (string) - Required - The serial number of the device.
#### Request Body
- **by** (string) - Required - The locator strategy (e.g., id, text, xpath).
- **value** (string) - Required - The value to search for.
- **timeout** (float) - Optional - The timeout in seconds for the click action.
### Request Example
```json
{
"by": "text",
"value": "Submit",
"timeout": 10.0
}
```
```
=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.