### Quick Start with AndroidAgent

Source: https://midscenejs.com/zh/android-api-reference

A quick start example demonstrating how to initialize an AndroidAgent, launch a website, perform an AI-driven action, and query for results.

```typescript
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';

const [first] = await getConnectedDevices();
const device = new AndroidDevice(first.udid);
await device.connect();

const agent = new AndroidAgent(device, {
  aiActionContext: 'If a permissions dialog appears, accept it.',
});

await agent.launch('https://www.ebay.com');
await agent.aiAct('search "Headphones" and wait for results');
const items = await agent.aiQuery(
  '{itemTitle: string, price: number}[], find item in list and corresponding price',
);
console.log(items);
```

--------------------------------

### Prepare and Run the Demo Project

Source: https://midscenejs.com/zh/integrate-with-any-interface

Clone the example project, navigate to the custom-interface directory, install dependencies, build the agent, and run the demo.

```bash
# 准备项目
git clone https://github.com/web-infra-dev/midscene-example.git
cd midscene-example/custom-interface
npm install
npm run build

# 运行演示
npm run demo
```

--------------------------------

### Quick Start Example

Source: https://midscenejs.com/zh/web-api-reference

Demonstrates a basic end-to-end flow using PlaywrightAgent to search for a product, extract information, and navigate.

```typescript
import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PlaywrightAgent(page);
await agent.aiAct('search "Noise cancelling headphones" and wait for results');
await agent.aiWaitFor('the results grid becomes visible');

const price = await agent.aiNumber('price of the first headphone');
console.log('first price', price);

await agent.aiTap('click the first result card');
await browser.close();
```

--------------------------------

### Quick Start with IOSAgent

Source: https://midscenejs.com/zh/ios-api-reference

Demonstrates a quick start for automating iOS devices using IOSDevice and IOSAgent. Ensure WDA is accessible and connected.

```typescript
import { IOSAgent, IOSDevice } from '@midscene/ios';

const device = new IOSDevice({ wdaHost: 'localhost', wdaPort: 8100 });
await device.connect();

const agent = new IOSAgent(device, {
  aiActionContext: 'If any permission dialog appears, accept it.',
});

await agent.launch('https://ebay.com');
await agent.aiAct('Search for "Headphones"');
const items = await agent.aiQuery(
  '{itemTitle: string, price: Number}[], list headphone products',
);
console.log(items);
```

--------------------------------

### TypeScript Example: Automate eBay Search and Assertion

Source: https://midscenejs.com/zh/android-getting-started

A comprehensive TypeScript example demonstrating how to use the AndroidAgent to open a browser, search for a product on eBay, query results, and assert page elements. It includes setup for environment variables and device connection.

```typescript
import 'dotenv/config'; // 通过 dotenv/config 自动加载 .env 文件中的环境变量
import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new AndroidDevice(devices[0].udid);

    const agent = new AndroidAgent(device, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await device.connect();

    await agent.aiAct('open browser and navigate to ebay.com');
    await sleep(5000);
    await agent.aiAct('type "Headphones" in search box, hit Enter');
    await agent.aiWaitFor('There is at least one headphone product');

    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    await agent.aiAssert('There is a category filter on the left');
  })(),
);

```

--------------------------------

### Install Xvfb and ImageMagick on Linux

Source: https://midscenejs.com/zh/computer-getting-started

Installs necessary packages for headless Linux environments to enable screenshots and interaction.

```bash
sudo apt-get install -y xvfb x11-xserver-utils imagemagick
```

--------------------------------

### Full Example: Vitest + HarmonyAgent for HarmonyOS Settings

Source: https://midscenejs.com/zh/harmony-getting-started

A comprehensive example demonstrating how to use Vitest with HarmonyAgent to test HarmonyOS settings, including toggling WLAN and Bluetooth. It sets up device connection, agent initialization, and custom test reporting.

```typescript
import type {
  TestStatus,
} from '@midscene/core';
import {
  ReportMergingTool,
} from '@midscene/core/report';
import {
  sleep,
} from '@midscene/core/utils';
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('HarmonyOS Settings Test', () => {
  let device: HarmonyDevice;
  let agent: HarmonyAgent;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    device = new HarmonyDevice(devices[0].deviceId);
    await device.connect();
  });

  beforeEach((ctx) => {
    agent = new HarmonyAgent(device, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-harmony-setting-test-report');
  });

  it('toggle WLAN', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('找到并进入 WLAN 设置');
    await agent.aiAct(
      '切换 WLAN 状态一次，如果 WLAN 关闭则打开，否则关闭。',
    );
  });

  it('toggle Bluetooth', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('找到并进入蓝牙设置');
    await agent.aiAct(
      '切换蓝牙状态一次，如果蓝牙关闭则打开，否则关闭。',
    );
  });
});
```

--------------------------------

### Computer Automation Example (macOS)

Source: https://midscenejs.com/zh/automate-with-scripts-in-yaml

Example script for computer automation on macOS, including opening applications, navigating URLs, and performing searches using aiAct.

```yaml
computer: {}

tasks:
  - name: 打开浏览器并搜索
    flow:
      - aiAct: 按下 Cmd+Space
      - sleep: 500
      - aiAct: 输入 "Safari" 并按回车
      - sleep: 2000
      - aiAct: 按下 Cmd+L 聚焦地址栏
      - aiAct: 输入 "https://www.bing.com"
      - aiAct: 按下回车
      - sleep: 3000
      - aiAct: 在搜索框中输入 "今日天气" 并按回车
      - aiAssert: 结果显示天气信息
```

--------------------------------

### Install Midscene Dependency

Source: https://midscenejs.com/zh/integrate-with-playwright

Install the Midscene web package as a development dependency.

```bash
npm install @midscene/web --save-dev
```

--------------------------------

### Full Example with Vitest

Source: https://midscenejs.com/zh/api

A comprehensive example demonstrating the integration of ReportMergingTool within a Vitest testing framework for Android automation.

```typescript
import { describe, it, beforeEach, afterEach, afterAll } from 'vitest';
import { AndroidAgent, AndroidDevice } from '@midscene/android';
import { ReportMergingTool } from '@midscene/core/report';

describe('Android 设置自动化', () => {
  let device: AndroidDevice;
  let agent: AndroidAgent;
  let startTime: number;
  const reportMergingTool = new ReportMergingTool();

  beforeEach((ctx) => {
    startTime = performance.now();
    agent = new AndroidAgent(device, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    // 确定自动化状态
    let workflowStatus = 'passed';
    if (ctx.task.result?.state === 'pass') {
      workflowStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      workflowStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0]?.message.includes('timed out')) {
      workflowStatus = 'timedOut';
    } else {
      workflowStatus = 'failed';
    }

    // 添加报告到合并列表
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: ctx.task.name,
        testTitle: ctx.task.name,
        testDescription: '自动化工作流描述',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: workflowStatus,
      },
    });
  });

  afterAll(() => {
    // 合并所有自动化报告
    reportMergingTool.mergeReports('android-settings-automation-report');
  });

  it('切换 WLAN', async () => {
    await agent.aiAct('找到并进入 WLAN 设置');
    await agent.aiAct('切换 WLAN 状态一次');
  });

  it('切换蓝牙', async () => {
    await agent.aiAct('找到并进入蓝牙设置');
    await agent.aiAct('切换蓝牙状态一次');
  });
});
```

--------------------------------

### Midscene iOS Agent Script Example

Source: https://midscenejs.com/zh/ios-getting-started

A TypeScript example demonstrating how to use the IOSAgent to automate interactions on an iOS device, including launching a browser, searching, querying data, and asserting conditions.

```typescript
import 'dotenv/config'; // 通过 dotenv/config 自动加载 .env 文件中的环境变量
import {
  IOSAgent,
  IOSDevice,
  agentFromWebDriverAgent,
} from '@midscene/ios';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    // 方式一：直接创建设备和 Agent
    const page = new IOSDevice({
      wdaPort: 8100,
      wdaHost: 'localhost',
    });

    // 👀 初始化 Midscene Agent
    const agent = new IOSAgent(page, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.',
    });
    await page.connect();

    // 方式二：使用便捷函数（推荐）
    // const agent = await agentFromWebDriverAgent({
    //   wdaPort: 8100,
    //   wdaHost: 'localhost',
    //   aiActionContext: 'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.',
    // });

    // 👀 直接打开 ebay.com（推荐做法）
    await page.launch('https://ebay.com');
    await sleep(3000);

    // 👀 输入关键字并执行搜索
    await agent.aiAct('Search for "Headphones"');

    // 👀 等待加载完成
    await agent.aiWaitFor('At least one headphone product is displayed on the page');
    // 或简单地等待几秒：
    // await sleep(5000);

    // 👀 理解页面内容并提取数据
    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find product titles and prices in the list',
    );
    console.log('Headphone product information', items);

    // 👀 使用 AI 断言
    await agent.aiAssert('Multiple headphone products are displayed on the interface');

    await page.destroy();
  })(),
);

```

--------------------------------

### Install Midscene Computer Package

Source: https://midscenejs.com/zh/computer-getting-started

Install the Midscene Computer package using your preferred package manager. This is the first step before writing your own scripts.

```bash
npm install @midscene/computer
```

--------------------------------

### Quick Start with HarmonyAgent

Source: https://midscenejs.com/zh/harmony-api-reference

Connect to a device, initialize a HarmonyAgent, and perform AI-driven actions like launching an app, scrolling, and querying UI elements.

```typescript
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const [first] = await getConnectedDevices();
const device = new HarmonyDevice(first.deviceId, {});
await device.connect();

const agent = new HarmonyAgent(device, {
  aiActionContext: '这是一台鸿蒙设备，如果出现弹窗，点击同意。',
});

await agent.launch('com.huawei.hmos.settings');
await agent.aiAct('scroll down one screen');
const items = await agent.aiQuery(
  'string[], list all visible setting item names',
);
console.log(items);
```

--------------------------------

### Start Playground CLI

Source: https://midscenejs.com/zh/android-getting-started

Launch the command-line interface for the Midscene Playground to test AI agent connections and flows without writing code.

```bash
npx --yes @midscene/android-playground
```

--------------------------------

### Launch Playground for an Agent

Source: https://midscenejs.com/zh/integrate-with-any-interface

This example demonstrates how to launch a Playground for a custom Agent. It includes setting the port, enabling automatic browser opening, and displaying server logs. The Playground is launched and then closed after a delay.

```typescript
import 'dotenv/config';
import { playgroundForAgent } from '@midscene/playground';
import { SampleDevice } from './sample-device';
import { Agent } from '@midscene/core/agent';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// 创建设备和 Agent 实例
const device = new SampleDevice();
const agent = new Agent(device);

// 启动 Playground
const result = await playgroundForAgent(agent).launch({
  port: 5800,
  openBrowser: true,
  verbose: true
});

console.log(`Playground 已启动：http://${result.host}:${result.port}`);

// 在需要时关闭 Playground
await sleep(10 * 60 * 1000); // 等待 10 分钟
await result.close();
console.log('Playground 已关闭！');
```

--------------------------------

### Launch Native App and Navigate

Source: https://midscenejs.com/zh/android-api-reference

Example of launching a native Android application and using agent actions to navigate back and go to the home screen.

```typescript
await agent.launch('com.android.settings/.Settings');
await agent.back();
await agent.home();
```

--------------------------------

### Start Playground CLI

Source: https://midscenejs.com/zh/computer-getting-started

Run this command to launch the Midscene Computer Playground CLI. This allows you to test connections and observe AI Agent behavior without writing code.

```bash
npx --yes @midscene/computer-playground
```

--------------------------------

### Install Dependencies for Playwright Integration

Source: https://midscenejs.com/zh/integrate-with-playwright

Install necessary packages including Midscene Web, Playwright, Playwright Test, and TSX for development.

```bash
npm install @midscene/web playwright @playwright/test tsx --save-dev
```

--------------------------------

### Write Your First Midscene Script

Source: https://midscenejs.com/zh/computer-getting-started

Create an example TypeScript file to interact with the Midscene Agent. This script demonstrates creating an agent, querying screen information, moving the mouse, and asserting screen content.

```typescript
import { agentForComputer } from '@midscene/computer';

(async () => {
  // 创建 agent
  const agent = await agentForComputer({
    aiActionContext: '你正在控制一台桌面计算机。',
  });

  // 截图并查询信息
  const screenInfo = await agent.aiQuery(
    '{width: number, height: number}, 获取屏幕分辨率'
  );
  console.log('屏幕分辨率:', screenInfo);

  // 移动鼠标到中心
  await agent.aiAct('将鼠标移动到屏幕中心');

  // 断言屏幕有内容
  await agent.aiAssert('屏幕有可见内容');

  console.log('桌面自动化完成！');
})();
```

--------------------------------

### Test Agent with Playground

Source: https://midscenejs.com/zh/integrate-with-any-interface

Launch a Playground service to test your Agent in a browser. This involves instantiating the device and agent, then starting the playground server.

```typescript
import 'dotenv/config'; // 从 .env 文件里读取 Midscene 环境变量
import { playgroundForAgent } from '@midscene/playground';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// 实例化 device 和 agent
const device = new SampleDevice();
await device.launch();
const agent = new Agent(device);

// 启动 playground
const server = await playgroundForAgent(agent).launch();

// 关闭 Playground
await sleep(10 * 60 * 1000);
await server.close();
console.log('Playground 已关闭！');

```

--------------------------------

### Connect to a New Chrome Tab

Source: https://midscenejs.com/zh/web-api-reference

Example of opening a new Chrome tab with a specified URL and performing AI actions.

```typescript
import { AgentOverChromeBridge } from '@midscene/web/bridge-mode';

const agent = new AgentOverChromeBridge();
await agent.connectNewTabWithUrl('https://www.bing.com');

await agent.ai('search "AI automation" and summarise first result');
await agent.aiAssert('some search results show up');
await agent.destroy();
```

--------------------------------

### Example Console Output for Headphone Information

Source: https://midscenejs.com/zh/integrate-with-puppeteer

This is the expected output when the demo script successfully extracts headphone data from the eBay page.

```json
[
  {
    "itemTitle": "Beats by Dr. Dre Studio Buds Totally Wireless Noise Cancelling In Ear + OPEN BOX",
    "price": 505.15
  },
  {
    "itemTitle": "Skullcandy Indy Truly Wireless Earbuds-Headphones Green Mint",
    "price": 186.69
  }
]
```

--------------------------------

### Run TypeScript Example

Source: https://midscenejs.com/zh/android-getting-started

Execute the compiled TypeScript demo script using tsx to run the automated browser interaction and agent tasks.

```bash
npx tsx demo.ts
```

--------------------------------

### Act: Fill Registration Form

Source: https://midscenejs.com/zh/android-getting-started

Example of using the 'Act' command to fill out a complete registration form, ensuring all fields pass validation.

```text
填写完整的注册表单，注意主要让所有字段通过校验
```

--------------------------------

### Tap: Click Login Button

Source: https://midscenejs.com/zh/android-getting-started

Example of using the 'Tap' command for an instant action to click a specific element on the page, like a login button.

```text
点击登录按钮
```

--------------------------------

### PlaywrightAgent Initialization and Usage

Source: https://midscenejs.com/zh/web-api-reference

Demonstrates how to import, initialize, and use the PlaywrightAgent for AI-driven actions within a Playwright browser context. Includes examples of searching, waiting for elements, and extracting data.

```APIDOC
## PlaywrightAgent

### Description

Use Midscene within a Playwright browser to support AI-powered testing or automation workflows.

### Import

```ts
import { PlaywrightAgent } from '@midscene/web/playwright';
```

### Constructor

```ts
const agent = new PlaywrightAgent(page, {
  // Browser-specific configurations...
});
```

### Browser-Specific Options

- `forceSameTabNavigation: boolean` - Forces execution within the current tab, defaults to `true`. Set to `false` to allow new tabs and create a new Agent for each.
- `waitForNavigationTimeout: number` - Timeout for waiting for navigation to complete, defaults to `5000` (0 to disable).
- `waitForNetworkIdleTimeout: number` - Timeout for waiting for network to be idle after each action, defaults to `2000` (0 to disable).
- `enableTouchEventsInActionSpace: boolean` - Enables touch gestures (like swipe) in the action space for pages requiring touch events, defaults to `false`.
- `keyboardTypeDelay: number` - Delay (in milliseconds) per character passed to Playwright's `page.keyboard.type`. Defaults to `undefined` (uses Playwright's default). Increase this value (e.g., `80`) if input fields miss characters during rapid typing.
- `forceChromeSelectRendering: boolean` - Forces `select` elements to use Chrome's base-select styling to prevent visibility issues with native system styles in screenshots or element extraction. Requires Playwright ≥ `1.52.0`. Defaults to `true`; set to `false` to disable (e.g., for older Chrome/Playwright versions).
- `customActions: DeviceAction[]` - Appends project-specific actions for the planner to invoke.

### Usage Notes

- One Agent per page: By default, `forceSameTabNavigation` is `true`, and Midscene intercepts new tabs for stability. Set to `false` and create a new Agent for each tab if new tabs are required.
- Refer to [API Reference (General)](./api#interaction-methods) for more interaction methods.

### Examples

#### Quick Start

```ts
import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PlaywrightAgent(page);
await agent.aiAct('search "Noise cancelling headphones" and wait for results');
await agent.aiWaitFor('the results grid becomes visible');

const price = await agent.aiNumber('price of the first headphone');
console.log('first price', price);

await agent.aiTap('click the first result card');
await browser.close();
```

#### Extending Playwright Tests with Midscene Fixture

```ts
// playwright.config.ts
export default defineConfig({
  reporter: [['list'], ['@midscene/web/playwright-reporter']],
});

// e2e/fixture.ts
import { test as base } from '@playwright/test';
import { PlaywrightAiFixture } from '@midscene/web/playwright';

export const test = base.extend(
  PlaywrightAiFixture({ waitForNetworkIdleTimeout: 1000 }),
);

// e2e/examples.spec.ts
test('search flow', async ({ agentForPage, page }) => {
  await page.goto('https://www.ebay.com');
  const agent = await agentForPage(page);
  await agent.aiAct('search "keyboard" and open first listing');
  await agent.aiAssert('a product detail page is opened');
});
```

This fixture also supports passing all configurations for `PlaywrightAgent`, allowing unified Agent behavior configuration during fixture creation. Meta-information automatically generated by tests, such as `testId`, `reportFileName`, `groupName`, and `groupDescription`, is still managed by the fixture.

### See Also

- [Integrate with Playwright](./integrate-with-playwright) for installation, fixture usage, and more configuration details.
```

--------------------------------

### Concurrent Execution with Error Handling

Source: https://midscenejs.com/zh/yaml-script-runner

This example demonstrates running multiple search scripts concurrently with error handling enabled, allowing execution to continue even if some scripts fail.

```bash
midscene --files './scripts/search-*.yaml' --concurrent 4 --continue-on-error
```

--------------------------------

### Launch Web or Native App

Source: https://midscenejs.com/zh/android-api-reference

Use the launch method to start a web URL or a native Android activity/package. The target can be a URL, package name, activity string, or an app name mapped in appNameMapping.

```typescript
function launch(target: string): Promise<void>;
```

--------------------------------

### Launch, Back, and Home Actions

Source: https://midscenejs.com/zh/harmony-api-reference

Demonstrates launching applications, navigating back, and returning to the home screen using the agent.

```typescript
await agent.launch('com.huawei.hmos.settings'); // 打开系统设置
await agent.launch('com.huawei.hmos.camera');    // 打开相机
await agent.back();
await agent.home();
```

--------------------------------

### Install Midscene Dependencies

Source: https://midscenejs.com/zh/ios-getting-started

Install the necessary Midscene iOS packages and dotenv for managing environment variables.

```bash
npm install @midscene/ios dotenv --save-dev
```

--------------------------------

### Quick Start with PuppeteerAgent

Source: https://midscenejs.com/zh/web-api-reference

Demonstrates launching a Puppeteer browser, creating a page, initializing PuppeteerAgent with an action context, performing AI actions like searching and querying, asserting conditions, and finally closing the browser.

```typescript
import puppeteer from 'puppeteer';
import { PuppeteerAgent } from '@midscene/web/puppeteer';

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PuppeteerAgent(page, {
  actionContext: 'When a cookie dialog appears, accept it.',
});

await agent.aiAct('search "Noise cancelling headphones" and open first result');
const items = await agent.aiQuery(
  '{itemTitle: string, price: number}[], list two products with price',
);
console.log(items);

await agent.aiAssert('there is a category filter on the left sidebar');
await browser.close();
```

--------------------------------

### Install Dependencies for Puppeteer Integration

Source: https://midscenejs.com/zh/integrate-with-puppeteer

Install necessary packages including @midscene/web, puppeteer, tsx, and dotenv for development.

```bash
npm install @midscene/web puppeteer tsx dotenv --save-dev
```

--------------------------------

### HarmonyDevice Constructor and Usage

Source: https://midscenejs.com/zh/harmony-api-reference

Demonstrates how to import and instantiate a HarmonyDevice, connect to it, and use it with a HarmonyAgent for automated tasks on HarmonyOS.

```APIDOC
## HarmonyDevice

Creates an HDC device instance that can be driven by HarmonyAgent.

### Import

```ts
import { HarmonyDevice, getConnectedDevices } from '@midscene/harmony';
```

### Constructor

```ts
const device = new HarmonyDevice(deviceId, {
  // Device options...
});
```

### Device Options

- `deviceId: string` - Value from `hdc list targets` or `getConnectedDevices()`.
- `hdcPath?: string` - Custom path to the HDC executable. If not set, it will be looked up in the `HDC_HOME` environment variable and common installation paths.
- `autoDismissKeyboard?: boolean` - Automatically dismiss the keyboard after input, defaults to `true`.
- `keyboardDismissStrategy?: 'esc-first' | 'back-first'` - Preferred key to use when automatically dismissing the keyboard, defaults to `'esc-first'`. HarmonyOS only sends the preferred key of this strategy: `'esc-first'` sends ESC, `'back-first'` sends Back.
- `screenshotResizeScale?: number` - **Deprecated.** This option is removed and no longer effective. To control the screenshot size sent to the AI model, use `screenshotShrinkFactor` in `AgentOpt`.
- `customActions?: DeviceAction[]` - Extend the planner's available actions via `defineAction`.

### Usage Notes

- `getConnectedDevices()` can be used to discover devices, returning `deviceId` consistent with `hdc list targets` output.
- If HDC is not in the system PATH, specify the path via the `HDC_HOME` environment variable or the `hdcPath` option.

### Example

#### Quick Start

```ts
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const [first] = await getConnectedDevices();
const device = new HarmonyDevice(first.deviceId, {});
await device.connect();

const agent = new HarmonyAgent(device, {
  aiActionContext: '这是一台鸿蒙设备，如果出现弹窗，点击同意。',
});

await agent.launch('com.huawei.hmos.settings');
await agent.aiAct('scroll down one screen');
const items = await agent.aiQuery(
  'string[], list all visible setting item names',
);
console.log(items);
```

#### Launch Application

```ts
await agent.launch('com.huawei.hmos.settings'); // Open system settings
await agent.launch('com.huawei.hmos.camera');    // Open camera
await agent.back();
await agent.home();
```
```

--------------------------------

### Configure Multi-Model Setup with Environment Variables

Source: https://midscenejs.com/zh/model-common-config

Set environment variables to configure the default multimodal model (Qwen 3.5), and separate models for planning and insight (GPT-5.4). Ensure you replace placeholder URLs and API keys with your actual credentials.

```bash
# Default multimodal model: Qwen 3.5
export MIDSCENE_MODEL_BASE_URL="https://..."       # Qwen 3.5 API address
export MIDSCENE_MODEL_API_KEY="..."                # Your Qwen 3.5 API Key
export MIDSCENE_MODEL_NAME="qwen3.5-plus"
export MIDSCENE_MODEL_FAMILY="qwen3.5"

# Planning model: GPT-5.4
export MIDSCENE_PLANNING_MODEL_API_KEY="sk-..."    # Your GPT-5.4 API Key
export MIDSCENE_PLANNING_MODEL_BASE_URL="https://..."
export MIDSCENE_PLANNING_MODEL_NAME="gpt-5.4"
export MIDSCENE_PLANNING_MODEL_FAMILY="gpt-5"

# Insight model: GPT-5.4
export MIDSCENE_INSIGHT_MODEL_API_KEY="sk-..."     # Your GPT-5.4 API Key
export MIDSCENE_INSIGHT_MODEL_BASE_URL="https://..."
export MIDSCENE_INSIGHT_MODEL_NAME="gpt-5.4"
export MIDSCENE_INSIGHT_MODEL_FAMILY="gpt-5"
```

--------------------------------

### Install Langfuse and OpenTelemetry Dependencies

Source: https://midscenejs.com/zh/model-config

Install the necessary npm packages for Langfuse integration and OpenTelemetry, which are required for tracing LLM calls with Langfuse.

```bash
npm install @langfuse/openai @langfuse/otel @opentelemetry/sdk-node
```

--------------------------------

### Skip Puppeteer Browser Download

Source: https://midscenejs.com/zh/integrate-with-puppeteer

Skip browser download during npm install to speed up dependency installation. Then, download Chrome separately from a mirror.

```bash
PUPPETEER_SKIP_DOWNLOAD=true npm install

npx puppeteer browsers install chrome --base-url="https://registry.npmmirror.com/-/binary/chrome-for-testing"
```

--------------------------------

### Install Midscene Android Dependencies

Source: https://midscenejs.com/zh/android-getting-started

Install the necessary Midscene Android agent and dotenv packages as development dependencies using npm or yarn.

```bash
npm install @midscene/android dotenv --save-dev
```

--------------------------------

### 使用 .env 文件配置环境变量

Source: https://midscenejs.com/zh/model-common-config

在项目根目录下创建 .env 文件，Midscene 命令行工具默认会读取此文件。

```dotenv
MIDSCENE_MODEL_BASE_URL="https://.../compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
MIDSCENE_MODEL_NAME="qwen3.7-plus"
MIDSCENE_MODEL_FAMILY="qwen3"
```

--------------------------------

### Install Langsmith Dependency

Source: https://midscenejs.com/zh/model-config

Install the Langsmith npm package to enable integration with the Langsmith LLM debugging platform. This is the first step in setting up Langsmith tracing.

```bash
npm install langsmith
```

--------------------------------

### TypeScript Equivalent of YAML Configuration

Source: https://midscenejs.com/zh/integrate-with-any-interface

This TypeScript code demonstrates how the YAML configuration translates into direct instantiation and agent setup. It shows importing the custom device class and initializing the agent with specific configurations.

```typescript
import MyDeviceClass from 'my-pkg-name';
const device = new MyDeviceClass();
const agent = new Agent(device, {
  output: './data.json',
});
```

--------------------------------

### Verify adb Installation

Source: https://midscenejs.com/zh/android-getting-started

Check if the Android Debug Bridge (adb) is installed correctly by running the version command. A successful output indicates that adb is properly set up on your system.

```bash
adb --version
```

--------------------------------

### Configure Qwen2.5-VL Model (Deprecated)

Source: https://midscenejs.com/zh/model-common-config

Configure environment variables for Qwen2.5-VL models. It is recommended to use newer Qwen3 models for better performance. Obtain the correct configuration from Alibaba Cloud.

```bash
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen-vl-max-latest"
MIDSCENE_MODEL_FAMILY="qwen2.5-vl"
```

--------------------------------

### 配置模型环境变量

Source: https://midscenejs.com/zh/skills

设置模型 API 密钥、名称、基础 URL 和家族标识符。这些变量可以作为系统环境变量设置，或放在当前工作目录的 .env 文件中。

```bash
MIDSCENE_MODEL_API_KEY="your-api-key"
MIDSCENE_MODEL_NAME="model-name"
MIDSCENE_MODEL_BASE_URL="https://..."
MIDSCENE_MODEL_FAMILY="family-identifier"
```

--------------------------------

### 千问 Qwen3.x 系列模型配置

Source: https://midscenejs.com/zh/model-common-config

配置阿里云的 Qwen3.x 系列模型，包括 API Key 和模型名称。

```bash
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen3.7-plus" # Qwen3.5 和 Qwen3.6 系列的 plus 模型分别为 "qwen3.5-plus"、"qwen3.6-plus"
MIDSCENE_MODEL_FAMILY="qwen3" # Qwen3.5 和 Qwen3.6 曾使用 `qwen3.5`、`qwen3.6`，目前仍然兼容
```

--------------------------------

### Terminate Specific App

Source: https://midscenejs.com/zh/android-api-reference

Example of terminating the 'com.android.settings' application.

```typescript
await agent.terminate('com.android.settings');
```

--------------------------------

### Get PageAgent Instance

Source: https://midscenejs.com/zh/integrate-with-playwright

Obtain a PageAgent instance from the agentForPage fixture to access lower-level capabilities.

```typescript
import { test } from './fixture';

test('case demo', async ({ agentForPage, page }) => {
  const agent = await agentForPage(page);

  await agent.recordToReport();
  const logContent = agent._unstableLogContent();
  console.log(logContent);
});
```

--------------------------------

### Define a Custom Action

Source: https://midscenejs.com/zh/integrate-with-any-interface

Example of defining a custom action named 'MyAction' with a string parameter 'name'.

```typescript
defineAction({
  name: 'MyAction',
  description: 'My action',
  paramSchema: z.object({
    name: z.string(),
  }),
  call: async (param) => {
    console.log(param.name);
  },
});
```

--------------------------------

### Initialize IOSAgent

Source: https://midscenejs.com/zh/ios-api-reference

Instantiate an IOSAgent with a device object and optional configuration. Common Agent parameters can be passed here.

```typescript
const agent = new IOSAgent(device, {
  // 通用 Agent 参数...
});
```

--------------------------------

### Configure package.json for CLI

Source: https://midscenejs.com/zh/integrate-with-any-interface

Add this 'bin' field to your package.json to make your CLI entry point executable. This allows AI assistants to run your custom interface using commands like 'npx my-device'.

```json
{
  "bin": {
    "my-device": "./dist/cli.js"
  }
}
```

--------------------------------

### Attach to Current Chrome Tab

Source: https://midscenejs.com/zh/web-api-reference

Example of attaching the agent to the currently active Chrome tab with specific configurations.

```typescript
import { AgentOverChromeBridge } from '@midscene/web/bridge-mode';

const agent = new AgentOverChromeBridge({
  allowRemoteAccess: false,
  closeNewTabsAfterDisconnect: true,
});

await agent.connectCurrentTab({ forceSameTabNavigation: true });
await agent.aiAct('open Gmail and report how many unread emails are visible');
await agent.destroy();
```

--------------------------------

### Execute adb Shell Tap Command

Source: https://midscenejs.com/zh/android-api-reference

Example of executing an 'input tap' command to simulate a touch event.

```typescript
await agent.runAdbShell('input tap 100 200');
```

--------------------------------

### Configure Qwen3-VL Model

Source: https://midscenejs.com/zh/model-common-config

Set these environment variables to use the Qwen3-VL model with Midscene.js. Ensure you have the correct base URL, API key, and model name from Alibaba Cloud.

```bash
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen3-vl-plus"
MIDSCENE_MODEL_FAMILY="qwen3-vl"
```

--------------------------------

### Execute and Log adb Shell Command

Source: https://midscenejs.com/zh/android-api-reference

Example of running a 'dumpsys battery' command with a timeout and logging the result.

```typescript
const result = await agent.runAdbShell('dumpsys battery', { timeout: 60 * 1000 });
console.log(result);
```

--------------------------------

### Example GitHub Registration Prompt

Source: https://midscenejs.com/zh/model-strategy

This prompt requires the AI to fill out a GitHub registration form, ensuring all fields are completed and validated, selecting 'United States' as the region, and returning the filled form data without initiating a real registration. It highlights the need for complex multi-step understanding and interaction.

```text
完成 GitHub 账号注册的表单填写，确保表单上没有遗漏的字段，选择“美国”作为地区。

确保所有的表单项能够通过校验。

只需要填写表单项即可，不需要发起真实的账号注册。

最终返回表单上实际填写的字段内容。
```

--------------------------------

### 配置环境变量的 .env 文件

Source: https://midscenejs.com/zh/yaml-script-runner

使用 .env 文件配置 Midscene 的模型服务地址、API Key、模型名称和系列。

```ini
MIDSCENE_MODEL_BASE_URL="https://替换为你的模型服务地址/v1"
MIDSCENE_MODEL_API_KEY="替换为你的 API Key"
MIDSCENE_MODEL_NAME="替换为你的模型名称"
MIDSCENE_MODEL_FAMILY="替换为你的模型系列"
```

--------------------------------

### Get Log Content

Source: https://midscenejs.com/zh/api

Retrieves log content from the report file. The structure of the log content may change in the future.

```typescript
const logContent = agent._unstableLogContent();
console.log(logContent);
```

--------------------------------

### 配置模型环境变量

Source: https://midscenejs.com/zh/quick-start

设置操作 Midscene 所需的多模态模型的基础 URL、API 密钥、模型名称和模型系列。示例使用阿里云的 Qwen3.x 模型。

```bash
export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export MIDSCENE_MODEL_API_KEY="your-api-key"
export MIDSCENE_MODEL_NAME="qwen3.7-plus"
export MIDSCENE_MODEL_FAMILY="qwen3"
```

--------------------------------

### Get Connected HDC Devices

Source: https://midscenejs.com/zh/harmony-api-reference

List all HDC devices that Midscene can control. This function can optionally take an HDC path.

```typescript
import { getConnectedDevices } from '@midscene/harmony';

const devices = await getConnectedDevices();
console.log(devices); // [{ deviceId: '0123456789ABCDEF' }]
```

--------------------------------

### 配置无头 Linux 模式

Source: https://midscenejs.com/zh/computer-getting-started

在无头 Linux 环境下运行 Midscene 时，可以通过传入 headless 选项或设置环境变量 MIDSCENE_COMPUTER_HEADLESS_LINUX=true 来启用无头模式。

```typescript
// 方式 1：传入 headless 选项
const agent = await agentForComputer({ headless: true });
```

```bash
// 方式 2：设置环境变量
// MIDSCENE_COMPUTER_HEADLESS_LINUX=true npx tsx example.ts
```

--------------------------------

### Get Connected Devices

Source: https://midscenejs.com/zh/android-api-reference

Retrieve a list of all adb devices that Midscene can control, including their unique ID, state, and port if applicable.

```typescript
function getConnectedDevices(): Promise<Array<{ 
  udid: string;
  state: string;
  port?: number;
}>>;
```

--------------------------------

### List and Connect to Specific Displays

Source: https://midscenejs.com/zh/computer-getting-started

Use `ComputerDevice.listDisplays()` to get information about available monitors and `agentForComputer` with `displayId` to connect to a specific one.

```typescript
import { ComputerDevice, agentForComputer } from '@midscene/computer';

// 列出所有显示器
const displays = await ComputerDevice.listDisplays();
console.log('可用显示器:', displays);

// 连接到特定显示器
const agent = await agentForComputer({
  displayId: displays[0].id,
});
```

--------------------------------

### Verify ANDROID_HOME Environment Variable

Source: https://midscenejs.com/zh/android-getting-started

Confirm that the ANDROID_HOME environment variable is set correctly. This variable is crucial for locating your Android SDK installation.

```bash
echo $ANDROID_HOME
```