### Project Setup and Demo Execution with npm Source: https://midscenejs.com/zh/integrate-with-any-interface Clone the Midscene example project, navigate to the custom interface directory, install dependencies, build the project, and run the demo. This provides a quick way to start and explore the features. ```bash # Prepare the project git clone https://github.com/web-infra-dev/midscene-example.git cd midscene-example/custom-interface npm install npm run build # Run the demo npm run demo ``` -------------------------------- ### iOS Getting Started with JavaScript SDK Source: https://midscenejs.com/zh/llms.txt Guide to controlling iOS devices using Midscene.js with JavaScript. It covers connecting to a real device via WebDriverAgent, configuring API keys, using the zero-code Playground, and running your first JavaScript script. Includes links to example projects for SDK demo and Vitest integration. ```javascript import { MidnodeClient } from "@midscene/midnode-client"; const client = new MidnodeClient({ // Configuration options }); async function runAutomation() { // Example: Launch an app await client.launchApp("com.example.app"); // Example: Interact with UI elements (requires visual model) await client.click({ text: "Login Button" }); // Example: Type text into an input field await client.type({ placeholder: "Username", text: "testuser" }); } runAutomation(); ``` -------------------------------- ### Manage Multi-Monitor Setup with Midscene Source: https://midscenejs.com/zh/computer-getting-started Example code for listing available displays and connecting the Midscene agent to a specific display using its ID. ```typescript import { ComputerDevice, agentFromComputer } from '@midscene/computer'; // 列出所有显示器 const displays = await ComputerDevice.listDisplays(); console.log('可用显示器:', displays); // 连接到特定显示器 const agent = await agentFromComputer({ displayId: displays[0].id, }); ``` -------------------------------- ### Prepare and Run Midscene Demo Project Source: https://context7_llms Clones the Midscene example project, navigates to the custom interface directory, installs dependencies, builds the project, and runs the demo. ```bash git clone https://github.com/web-infra-dev/midscene-example.git cd midscene-example/custom-interface npm install npm run build npm run demo ``` -------------------------------- ### Configure WebDriver for iOS Automation Source: https://midscenejs.com/zh/changelog This example shows the basic setup for connecting to an iOS device using WebDriver. It's a prerequisite for enabling iOS automation capabilities within Midscene. ```javascript const iosAgent = await createIOSAgent({ "platformName": "iOS", "appium:platformVersion": "16.0", "appium:deviceName": "iPhone 14", "appium:automationName": "XCUITest", "appium:app": "/path/to/your.app" }); ``` -------------------------------- ### Quick Start: Launch Website and AI Act with IOSAgent Source: https://midscenejs.com/zh/ios-api-reference Demonstrates the basic setup for using IOSAgent to connect to an iOS device, launch a website, perform an AI-driven search action, and query for results. Requires `@midscene/ios` package. ```typescript import { IOSAgent, IOSDevice } from '@midscene/ios'; const device = new IOSDevice({ wdaHost: 'localhost', wdaPort: 8100 }); await device.connect(); const agent = new IOSAgent(device, { aiActionContext: 'If any permission dialog appears, accept it.', }); await agent.launch('https://ebay.com'); await agent.aiAct('Search for "Headphones"'); const items = await agent.aiQuery( '{itemTitle: string, price: Number}[], list headphone products', ); console.log(items); ``` -------------------------------- ### Multi-Display Workflow Example Source: https://midscenejs.com/zh/computer-api-reference Demonstrates a workflow involving multiple displays. It shows how to list all displays and then create separate ComputerAgent instances to control actions on each display independently, useful for managing complex screen setups. ```typescript import { ComputerDevice, agentFromComputer } from '@midscene/computer'; // List displays const displays = await ComputerDevice.listDisplays(); console.log(`找到 ${displays.length} 个显示器`); // Control primary display const agent1 = await agentFromComputer({ displayId: displays[0].id, }); await agent1.aiAct('将鼠标移动到屏幕中心'); // Control secondary display if available if (displays.length > 1) { const agent2 = await agentFromComputer({ displayId: displays[1].id, }); await agent2.aiAct('将鼠标移动到屏幕中心'); } ``` -------------------------------- ### Troubleshooting LLM Connectivity with Example Project Source: https://context7_llms Provides instructions for troubleshooting connection issues with model services. It involves using the 'connectivity-test' example project, placing a .env file in the specified folder, and running npm install and npm run test. ```text 将你的 .env 文件放在 `connectivity-test` 文件夹中,然后运行 `npm i && npm run test` 来进行测试。 ``` -------------------------------- ### Computer (Desktop) Usage Example Source: https://midscenejs.com/zh/automate-with-scripts-in-yaml Example demonstrating desktop automation tasks. ```APIDOC ## Computer (Desktop) Usage Example ### Description Example demonstrating desktop automation tasks, including launching applications, typing, and asserting screen content. ### Request Example ```yaml computer: {} tasks: - name: Open Browser and Search flow: - aiAct: Press Cmd+Space # On Windows, use 'Press Windows key' - sleep: 500 - aiAct: Type "Safari" and press Enter # On Windows, use 'Type "Chrome"' - sleep: 2000 - aiAct: Press Cmd+L to focus address bar # On Windows, use 'Press Ctrl+L' - aiAct: Type "https://www.example.com" - aiAct: Press Enter - sleep: 3000 - aiAct: Type "Weather today" in search box and press Enter - aiAssert: Results show weather information ``` ### Platform Notes - **macOS**: Use `Cmd+Space`, `Safari`, `Cmd+L` as shown in the example. - **Windows**: Replace `Cmd+Space` with `Press Windows key`, `Safari` with `Chrome`, and `Cmd+L` with `Ctrl+L`. ``` -------------------------------- ### Automate Desktop Actions (Windows Example) Source: https://midscenejs.com/zh/automate-with-scripts-in-yaml Example of automating desktop tasks on Windows, demonstrating equivalent actions to the macOS example. It involves using Windows key shortcuts and adapting application names. ```yaml # For Windows, modify the script: # Use '按下 Windows 键' instead of '按下 Cmd+Space' # Use '输入 "Chrome"' instead of '输入 "Safari"' # Use '按下 Ctrl+L' instead of '按下 Cmd+L' ``` -------------------------------- ### PC Desktop Automation (JavaScript SDK Example) Source: https://midscenejs.com/zh/changelog Shows an example of PC desktop automation using the JavaScript SDK. This approach allows developers to script mouse and keyboard actions, capture screenshots, and interact with desktop applications programmatically on Windows, macOS, and Linux. ```javascript // Example JavaScript for PC Desktop Automation await computer.click({ x: 100, y: 200 }); await computer.typeText('Hello, Desktop!'); await computer.captureScreen(); ``` -------------------------------- ### Install @midscene/computer Dependency Source: https://midscenejs.com/zh/computer-getting-started Instructions for installing the @midscene/computer package using various package managers like npm, yarn, pnpm, bun, and deno. ```bash npm install @midscene/computer ``` ```bash yarn add @midscene/computer ``` ```bash pnpm add @midscene/computer ``` ```bash bun add @midscene/computer ``` ```bash deno add npm:@midscene/computer ``` -------------------------------- ### Example: Automate Safari Search with Midscene iOS Agent Source: https://midscenejs.com/zh/ios-getting-started A comprehensive JavaScript example demonstrating how to use the Midscene iOS Agent to launch Safari, search for a product, extract data, and perform assertions. It includes agent initialization, page navigation, and AI-driven actions. ```TypeScript import { IOSAgent, IOSDevice, agentFromWebDriverAgent, } from '@midscene/ios'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { // 方式一:直接创建设备和 Agent const page = new IOSDevice({ wdaPort: 8100, wdaHost: 'localhost', }); // 👀 初始化 Midscene Agent const agent = new IOSAgent(page, { aiActionContext: 'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.', }); await page.connect(); // 方式二:使用便捷函数(推荐) // const agent = await agentFromWebDriverAgent({ // wdaPort: 8100, // wdaHost: 'localhost', // aiActionContext: 'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.', // }); // 👀 直接打开 ebay.com(推荐做法) await page.launch('https://ebay.com'); await sleep(3000); // 👀 输入关键字并执行搜索 await agent.aiAct('Search for "Headphones"'); // 👀 等待加载完成 await agent.aiWaitFor('At least one headphone product is displayed on the page'); // 或简单地等待几秒: // await sleep(5000); // 👀 理解页面内容并提取数据 const items = await agent.aiQuery( '{itemTitle: string, price: Number}[], find product titles and prices in the list', ); console.log('Headphone product information', items); // 👀 使用 AI 断言 await agent.aiAssert('Multiple headphone products are displayed on the interface'); await page.destroy(); })(), ); ``` -------------------------------- ### Quick Start with Playwright Agent Source: https://midscenejs.com/zh/web-api-reference Illustrates initializing a PlaywrightAgent for AI-driven automation within a Playwright browser context. This example demonstrates searching, waiting for results, extracting numerical data, and interacting with elements. It requires Playwright and the @midscene/web/playwright package. ```javascript import { chromium } from 'playwright'; import { PlaywrightAgent } from '@midscene/web/playwright'; const browser = await chromium.launch({ headless: true }); const page = await browser.newPage(); await page.goto('https://www.ebay.com'); const agent = new PlaywrightAgent(page); await agent.aiAct('search "Noise cancelling headphones" and wait for results'); await agent.aiWaitFor('the results grid becomes visible'); const price = await agent.aiNumber('price of the first headphone'); console.log('first price', price); await agent.aiTap('click the first result card'); await browser.close(); ``` -------------------------------- ### Example Desktop Automation Script (macOS) Source: https://midscenejs.com/zh/automate-with-scripts-in-yaml.html An example YAML script demonstrating desktop automation tasks on macOS, including launching applications, typing, and assertions. ```yaml computer: {} tasks: - name: 打开浏览器并搜索 flow: - aiAct: 按下 Cmd+Space - sleep: 500 - aiAct: 输入 "Safari" 并按回车 - sleep: 2000 - aiAct: 按下 Cmd+L 聚焦地址栏 - aiAct: 输入 "https://www.bing.com" - aiAct: 按下回车 - sleep: 3000 - aiAct: 在搜索框中输入 "今日天气" 并按回车 - aiAssert: 结果显示天气信息 ``` -------------------------------- ### Example Desktop Automation Script (Windows) Source: https://midscenejs.com/zh/automate-with-scripts-in-yaml.html An example YAML script demonstrating desktop automation tasks on Windows, including launching applications, typing, and assertions. Note the platform-specific key commands. ```yaml computer: {} tasks: - name: 打开浏览器并搜索 flow: - aiAct: 按下 Windows 键 - sleep: 500 - aiAct: 输入 "Chrome" 并按回车 - sleep: 2000 - aiAct: 按下 Ctrl+L 聚焦地址栏 - aiAct: 输入 "https://www.bing.com" - aiAct: 按下回车 - sleep: 3000 - aiAct: 在搜索框中输入 "今日天气" 并按回车 - aiAssert: 结果显示天气信息 ``` -------------------------------- ### Install Midscene Android Agent Dependencies Source: https://midscenejs.com/zh/android-getting-started These commands show how to install the `@midscene/android` package as a development dependency using different package managers (npm, yarn, pnpm, bun, deno). ```bash npm install @midscene/android --save-dev ``` ```bash yarn add @midscene/android --save-dev ``` ```bash pnpm add @midscene/android --save-dev ``` ```bash bun add @midscene/android --save-dev ``` ```bash deno add npm:@midscene/android --save-dev ``` -------------------------------- ### Troubleshoot Model Service Connectivity with Example Project Source: https://context7_llms Use the 'connectivity-test' example project to troubleshoot model service connectivity issues. Place your `.env` file in the `connectivity-test` folder and run `npm i && npm run test`. ```bash cd connectivity-test npm i && npm run test ``` -------------------------------- ### AndroidDevice Example Source: https://midscenejs.com/zh/llms-full.txt A practical example demonstrating how to connect to an Android device, create an agent, and perform automated actions. ```APIDOC ### Example #### Quick Start ```ts import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android'; const [first] = await getConnectedDevices(); const device = new AndroidDevice(first.udid); await device.connect(); const agent = new AndroidAgent(device, { aiActionContext: 'If a permissions dialog appears, accept it.', }); await agent.launch('https://www.ebay.com'); await agent.aiAct('search "Headphones" and wait for results'); const items = await agent.aiQuery( '{itemTitle: string, price: number}[], find item in list and corresponding price', ); console.log(items); ``` #### Launch Native App ```ts await agent.launch('com.android.settings/.Settings'); await agent.back(); await agent.home(); ``` ``` -------------------------------- ### Midscene iOS Automation Script Example Source: https://midscenejs.com/zh/integrate-with-ios A TypeScript example demonstrating how to use Midscene's IOSAgent and IOSDevice to automate tasks in iOS Safari. It includes initializing the agent, launching a URL, performing AI actions like searching and querying, and asserting conditions. ```TypeScript import { IOSAgent, IOSDevice, agentFromWebDriverAgent, } from '@midscene/ios'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { // 方法一:直接创建设备和 Agent const page = new IOSDevice({ wdaPort: 8100, wdaHost: 'localhost', }); // 👀 初始化 Midscene agent const agent = new IOSAgent(page, { aiActionContext: '如果出现位置、权限、用户协议等弹窗,点击同意。如果出现登录页面,关闭它。', }); await page.connect(); // 方法二:或者使用便捷函数(推荐) // const agent = await agentFromWebDriverAgent({ // wdaPort: 8100, // wdaHost: 'localhost', // aiActionContext: '如果出现位置、权限、用户协议等弹窗,点击同意。如果出现登录页面,关闭它。', // }); // 👀 打开 ebay.com 网页 await page.launch('https://ebay.com'); await sleep(3000); // 👀 输入关键词,执行搜索 await agent.aiAct('在搜索框输入 "Headphones",敲回车'); // 👀 等待加载完成 await agent.aiWaitFor('页面中至少有一个耳机商品'); // 或者你也可以使用一个普通的 sleep: // await sleep(5000); // 👀 理解页面内容,提取数据 const items = await agent.aiQuery( '{itemTitle: string, price: Number}[], 找到列表里的商品标题和价格', ); console.log('耳机商品信息', items); // 👀 用 AI 断言 await agent.aiAssert('界面中有多个耳机产品'); await page.destroy(); })(), ); ``` -------------------------------- ### Open Application and Navigate Example Source: https://midscenejs.com/zh/computer-api-reference A comprehensive example demonstrating how to use the ComputerAgent to open an application (like TextEdit or Notepad) and perform subsequent actions such as typing text and saving a file. It includes platform-specific logic for macOS and Windows. ```typescript import { agentFromComputer } from '@midscene/computer'; const agent = await agentFromComputer(); // Open application (platform-specific) if (process.platform === 'darwin') { await agent.aiAct('按 Cmd+Space'); await agent.aiAct('输入 "文本编辑" 并按回车'); } else { await agent.aiAct('按 Windows 键'); await agent.aiAct('输入 "记事本" 并按回车'); } await agent.aiWaitFor('文本编辑器窗口可见'); // Input text await agent.aiAct('输入 "你好,Midscene!"'); // Save file (platform-specific) if (process.platform === 'darwin') { await agent.aiAct('按 Cmd+S'); } else { await agent.aiAct('按 Ctrl+S'); } ``` -------------------------------- ### PC Desktop Automation (YAML Example) Source: https://midscenejs.com/zh/changelog Illustrates the use of YAML for PC desktop automation, a core feature introduced in v1.3. This format allows for defining automation scripts for native keyboard and mouse operations across Windows, macOS, and Linux, supporting various application types like Electron, Qt, and WPF. ```yaml # Example YAML for PC Desktop Automation - action: click target: "button[text='Submit']" - action: type target: "input[name='username']" value: "testuser" - action: keyPress keys: ["Ctrl", "C"] ``` -------------------------------- ### Run Midscene Android Agent Script Source: https://midscenejs.com/zh/android-getting-started This command executes a TypeScript file (e.g., `demo.ts`) using `tsx`, a JavaScript runtime that supports TypeScript out of the box. This is used to run the example script after installation. ```bash npx tsx demo.ts ``` -------------------------------- ### Example Usage of aiAsk in TypeScript Source: https://midscenejs.com/zh/llms-full.txt Illustrates how to use the aiAsk function in TypeScript to get an answer from the AI model based on a natural language question about the current page. ```typescript const result = await agent.aiAsk('当前页面的应该怎么进行测试?'); console.log(result); // Output the AI model's answer ``` -------------------------------- ### Start Local HTTP Server with Node.js Source: https://midscenejs.com/zh/api.html This command starts a simple HTTP server in the current directory using Node.js's `npx serve`. This is useful for locally testing HTML reports generated with the `'html-and-external-assets'` output format, which require a server to load external assets like images due to browser CORS restrictions when accessed via `file://`. ```bash npx serve ``` -------------------------------- ### Initialize IOSAgent with agentFromWebDriverAgent() Source: https://midscenejs.com/zh/integrate-with-ios Creates an IOSAgent instance by connecting to a WebDriverAgent service. This is the recommended and simplest method for agent initialization. It accepts optional configuration options for the agent and device. ```typescript import { agentFromWebDriverAgent } from '@midscene/ios'; // Use default WebDriverAgent address (localhost:8100) const agent = await agentFromWebDriverAgent(); // Use custom WebDriverAgent address const agent = await agentFromWebDriverAgent({ wdaHost: 'localhost', wdaPort: 8100, aiActionContext: 'If a popup appears, click agree', }); ``` -------------------------------- ### Example Midscene Android Agent Script Source: https://midscenejs.com/zh/android-getting-started This TypeScript script demonstrates how to use the Midscene Android Agent to open a browser, navigate to a website, perform a search, query for specific data, and make an assertion about the page content. It includes setup for connecting to a device and configuring the agent. ```typescript import { AndroidAgent, AndroidDevice, getConnectedDevices, } from '@midscene/android'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { const devices = await getConnectedDevices(); const device = new AndroidDevice(devices[0].udid); const agent = new AndroidAgent(device, { aiActionContext: 'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.', }); await device.connect(); await agent.aiAct('open browser and navigate to ebay.com'); await sleep(5000); await agent.aiAct('type "Headphones" in search box, hit Enter'); await agent.aiWaitFor('There is at least one headphone product'); const items = await agent.aiQuery( '{itemTitle: string, price: Number}[], find item in list and corresponding price', ); console.log('headphones in stock', items); await agent.aiAssert('There is a category filter on the left'); })(), ); ``` -------------------------------- ### Install Midscene Web and Puppeteer Dependencies Source: https://context7_llms Install the necessary npm packages for integrating Midscene with Puppeteer. This command installs `@midscene/web`, `puppeteer`, and `tsx` as development dependencies. ```bash npm install @midscene/web puppeteer tsx --save-dev ``` -------------------------------- ### Start Midscene Playground CLI Source: https://midscenejs.com/zh/android-getting-started This command initiates the Midscene Playground command-line interface. The Playground provides a zero-code environment to test connections and observe AI Agent behavior, sharing the same core logic as the `@midscene/android` package. ```bash npx --yes @midscene/android-playground ``` -------------------------------- ### YAML Script Runner Example Source: https://midscenejs.com/zh/llms.txt Demonstrates how to use Midscene.js's YAML format for automation scripts and its command-line tool for execution. This simplifies environment configuration and provides a visual report upon completion. The YAML format allows for quick script writing. ```yaml name: "Example Web Automation" steps: - action: "open" url: "https://www.example.com" - action: "click" selector: "#myButton" - action: "type" selector: "#myInput" text: "Hello Midscene!" - action: "screenshot" path: "./screenshots/final.png" ``` ```bash midscene run --file ./automation.yaml ``` -------------------------------- ### Install Dependencies for Remote Puppeteer Integration Source: https://midscenejs.com/zh/llms-full.txt Install Puppeteer and the Midscene web package as development dependencies, which are required for connecting to a remote Puppeteer browser and integrating it with Midscene Agent. ```bash npm install puppeteer @midscene/web --save-dev ``` -------------------------------- ### Install Midscene.js Web Package for Playwright Source: https://midscenejs.com/zh/integrate-with-playwright These commands show how to install the `@midscene/web` package as a development dependency for Playwright using different package managers (npm, yarn, pnpm, bun, deno). ```bash npm install @midscene/web --save-dev ``` ```bash yarn add @midscene/web --save-dev ``` ```bash pnpm add @midscene/web --save-dev ``` ```bash bun add @midscene/web --save-dev ``` ```bash deno add npm:@midscene/web --save-dev ``` -------------------------------- ### Use Qwen3-VL Model for Visual Understanding Source: https://midscenejs.com/zh/changelog This configuration snippet indicates the use of the Qwen3-VL model, a recent advancement in visual understanding capabilities. It aims to provide faster and more accurate visual comprehension for automation tasks. ```javascript const agent = await createAgent({ model: 'qwen3-vl' }); ``` -------------------------------- ### Install Midscene iOS Agent Dependencies Source: https://midscenejs.com/zh/ios-getting-started Instructions for installing the Midscene iOS agent using various package managers like npm, yarn, pnpm, bun, and deno. This is a prerequisite for integrating the agent into your JavaScript projects. ```npm npm install @midscene/ios --save-dev ``` ```yarn yarn add @midscene/ios --save-dev ``` ```pnpm pnpm add @midscene/ios --save-dev ``` ```bun bun add @midscene/ios --save-dev ``` ```deno deno add npm:@midscene/ios --save-dev ``` -------------------------------- ### Example: Assert Page Element with Image Source: https://midscenejs.com/zh/llms-full.txt Illustrates how to use an image as part of an assertion to verify the presence of a specific element on a page. This example uses the `agent.aiAssert` method with a prompt and an image. ```javascript await agent.aiAssert({ prompt: 'Is the specified logo present on the page?', images: [ { name: 'Specify logo', url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png', }, ], }); ``` -------------------------------- ### Start Local HTTP Server with Python 3 Source: https://midscenejs.com/zh/api.html This command starts a simple HTTP server in the current directory using Python 3's built-in http.server module. This is useful for locally testing HTML reports generated with the `'html-and-external-assets'` output format, which require a server to load external assets like images due to browser CORS restrictions when accessed via `file://`. ```bash python3 -m http.server ``` -------------------------------- ### Create First Midscene Agent Script Source: https://midscenejs.com/zh/computer-getting-started A basic TypeScript example demonstrating how to initialize a Midscene agent, query screen information, perform an action, and assert screen content. ```typescript import { agentFromComputer } from '@midscene/computer'; (async () => { // 创建 agent const agent = await agentFromComputer({ aiActionContext: '你正在控制一台桌面计算机。', }); // 截图并查询信息 const screenInfo = await agent.aiQuery( '{width: number, height: number}, 获取屏幕分辨率' ); console.log('屏幕分辨率:', screenInfo); // 移动鼠标到中心 await agent.aiAct('将鼠标移动到屏幕中心'); // 断言屏幕有内容 await agent.aiAssert('屏幕有可见内容'); console.log('桌面自动化完成!'); })(); ``` -------------------------------- ### Example Prompts for Midscene.js Interface Interactions Source: https://context7_llms Examples of prompts for interacting with web interfaces using Midscene.js. These demonstrate capabilities like form filling, data extraction, and element assertion. ```plaintext Sign up for Github, you need to pass the form validation, but don't actually click. ``` ```plaintext In the search box, enter Midscene, perform the search, and navigate to the first result. ``` ```plaintext Fill in the complete registration form, ensuring all fields pass validation. ``` ```plaintext Extract the user ID from the page, returning JSON data structured as { id: string }. ``` ```plaintext There is a login button on the page, and below it is a link to the user agreement. ``` ```plaintext Click the login button. ``` -------------------------------- ### Install Midscene Android Package Source: https://context7_llms Installs the Midscene package specifically for Android automation as a development dependency. This package provides the necessary tools and SDK components for controlling Android devices with Midscene. ```bash npm install @midscene/android --save-dev ```