### Quick Start with IOSAgent

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/ios-api-reference.mdx

Initialize IOSDevice and IOSAgent, then launch a website and perform an AI-driven search. This example demonstrates basic setup and interaction.

```typescript
import { IOSAgent, IOSDevice } from '@midscene/ios';

const device = new IOSDevice({ wdaHost: 'localhost', wdaPort: 8100 });
await device.connect();

const agent = new IOSAgent(device, {
  aiActionContext: 'If any permission dialog appears, accept it.',
});

await agent.launch('https://ebay.com');
await agent.aiAct('Search for "Headphones"');
const items = await agent.aiQuery(
  '{itemTitle: string, price: Number}[], list headphone products',
);
console.log(items);
```

--------------------------------

### PuppeteerAgent Quick Start Example

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/web-api-reference.mdx

A basic example demonstrating how to initialize and use PuppeteerAgent for AI-driven actions and queries.

```APIDOC
## PuppeteerAgent Quick Start Example

### Description
This example shows how to launch a Puppeteer browser, create a `PuppeteerAgent`, perform an AI action, query for data, and assert a condition.

### Usage
```ts
import puppeteer from 'puppeteer';
import { PuppeteerAgent } from '@midscene/web/puppeteer';

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PuppeteerAgent(page, {
  actionContext: 'When a cookie dialog appears, accept it.',
});

await agent.aiAct('search "Noise cancelling headphones" and open first result');
const items = await agent.aiQuery(
  '{itemTitle: string, price: number}[], list two products with price',
);
console.log(items);

await agent.aiAssert('there is a category filter on the left sidebar');
await browser.close();
```
```

--------------------------------

### Install Node.js 20.9.0 with nvm

Source: https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md

Example commands to install a specific Node.js version using nvm, set it as default, and switch to it.

```bash
# Install the LTS version of Node.js 20
nvm install 20.9.0 --lts

# Make the newly installed Node.js 20 as the default version
nvm alias default 20.9.0

# Switch to the newly installed Node.js 20
nvm use 20.9.0
```

--------------------------------

### Install Midscene CLI Globally

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/yaml-script-runner.mdx

Command to install the Midscene CLI globally using npm, recommended for initial setup.

```bash
npm i -g @midscene/cli
```

--------------------------------

### Complete Example: Vitest + AndroidAgent

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/android-getting-started.mdx

A comprehensive example demonstrating how to set up an Android testing environment using Vitest and AndroidAgent. It includes setup, teardown, and test cases for toggling WLAN and Bluetooth settings.

```typescript
import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';
import type { TestStatus } from '@midscene/core';
import { ReportMergingTool } from '@midscene/core/report';
import { sleep } from '@midscene/core/utils';
import type ADB from 'appium-adb';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('Android Settings Test', () => {
  let page: AndroidDevice;
  let adb: ADB;
  let agent: AndroidAgent;
  let startTime: number;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    page = new AndroidDevice(devices[0].udid);
    adb = await page.getAdb();
  });

  beforeEach((ctx) => {
    startTime = performance.now();
    agent = new AndroidAgent(page, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-android-setting-test-report');
  });

  it('toggle wlan', async () => {
    await adb.shell('input keyevent KEYCODE_HOME');
    await sleep(1000);
    await adb.shell('am start -n com.android.settings/.Settings');
    await sleep(1000);
    await agent.aiAct('find and enter WLAN setting');
    await agent.aiAct(
      'toggle WLAN status *once*, if WLAN is off pls turn it on, otherwise turn it off.',
    );
  });

  it('toggle bluetooth', async () => {
    await adb.shell('input keyevent KEYCODE_HOME');
    await sleep(1000);
    await adb.shell('am start -n com.android.settings/.Settings');
    await sleep(1000);
    await agent.aiAct('find and enter bluetooth setting');
    await agent.aiAct(
      'toggle bluetooth status *once*, if bluetooth is off pls turn it on, otherwise turn it off.',
    );
  });
});
```

--------------------------------

### Quick Start with PlaywrightAgent

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/web-api-reference.mdx

A complete example demonstrating launching Playwright, navigating to a page, and using the agent for AI-driven actions like searching, waiting, and extracting data.

```typescript
import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PlaywrightAgent(page);
await agent.aiAct('search "Noise cancelling headphones" and wait for results');
await agent.aiWaitFor('the results grid becomes visible');

const price = await agent.aiNumber('price of the first headphone');
console.log('first price', price);

await agent.aiTap('click the first result card');
await browser.close();
```

--------------------------------

### Quick Start Android Agent

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/android-api-reference.mdx

Connect to a device, launch a website, perform an AI-driven search, and query results. Use this for initial setup and basic automation tasks.

```typescript
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';

const [first] = await getConnectedDevices();
const device = new AndroidDevice(first.udid);
await device.connect();

const agent = new AndroidAgent(device, {
  aiActionContext: 'If a permissions dialog appears, accept it.',
});

await agent.launch('https://www.ebay.com');
await agent.aiAct('search "Headphones" and wait for results');
const items = await agent.aiQuery(
  '{itemTitle: string, price: number}[], find item in list and corresponding price',
);
console.log(items);
```

--------------------------------

### Complete Vitest + HarmonyAgent Example for HarmonyOS

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/harmony-getting-started.mdx

A comprehensive example demonstrating how to set up Vitest for testing HarmonyOS applications using `HarmonyAgent`. It includes setup for connecting devices, managing agent instances, and reporting test results.

```typescript
import type { TestStatus } from '@midscene/core';
import { ReportMergingTool } from '@midscene/core/report';
import { sleep } from '@midscene/core/utils';
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('HarmonyOS Settings Test', () => {
  let device: HarmonyDevice;
  let agent: HarmonyAgent;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    device = new HarmonyDevice(devices[0].deviceId);
    await device.connect();
  });

  beforeEach((ctx) => {
    agent = new HarmonyAgent(device, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-harmony-setting-test-report');
  });

  it('toggle WLAN', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('find and enter WLAN setting');
    await agent.aiAct(
      'toggle WLAN status *once*, if WLAN is off please turn it on, otherwise turn it off.',
    );
  });

  it('toggle Bluetooth', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('find and enter Bluetooth setting');
    await agent.aiAct(
      'toggle Bluetooth status *once*, if Bluetooth is off please turn it on, otherwise turn it off.',
    );
  });
});
```

--------------------------------

### Install Midscene Computer SDK

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/computer-getting-started.mdx

Install the @midscene/computer package using your preferred package manager. This command is for general installation.

```bash
npm install @midscene/computer
```

--------------------------------

### Configure Multi-model Setup (GPT-5.4 + Qwen 3.5)

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/model-common-config.mdx

Configure environment variables for a multi-model setup using GPT-5.4 for planning/insight and Qwen 3.5 for visual grounding. This allows leveraging heavy reasoning from GPT-5.4 while Qwen 3.5 handles visual tasks.

```bash
# Default vision model: Qwen 3.5
export MIDSCENE_MODEL_BASE_URL="https://..."       # Qwen 3.5 endpoint
export MIDSCENE_MODEL_API_KEY="..."                # Your Qwen 3.5 API key
export MIDSCENE_MODEL_NAME="qwen3.5-plus"
export MIDSCENE_MODEL_FAMILY="qwen3.5"

# Planning model: GPT-5.4
export MIDSCENE_PLANNING_MODEL_API_KEY="sk-..."    # Your GPT-5.4 API key
export MIDSCENE_PLANNING_MODEL_BASE_URL="https://..." 
export MIDSCENE_PLANNING_MODEL_NAME="gpt-5.4"
export MIDSCENE_PLANNING_MODEL_FAMILY="gpt-5"

# Insight model: GPT-5.4
export MIDSCENE_INSIGHT_MODEL_API_KEY="sk-..."     # Your GPT-5.4 API key
export MIDSCENE_INSIGHT_MODEL_BASE_URL="https://..."
export MIDSCENE_INSIGHT_MODEL_NAME="gpt-5.4"
export MIDSCENE_INSIGHT_MODEL_FAMILY="gpt-5"
```

--------------------------------

### MCP Server Configuration

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/mcp-android.mdx

Configuration example for adding the Midscene MCP server to your MCP setup, including required environment variables.

```APIDOC
## MCP Server Configuration

Add the Midscene MCP server to your MCP configuration. Note that the `MIDSCENE_MCP_ANDROID_MODE` environment variable is required.

```json
{
  "mcpServers": {
    "mcp-midscene": {
      "command": "npx",
      "args": ["-y", "@midscene/mcp"],
      "env": {
        "MIDSCENE_MODEL_NAME": "REPLACE_WITH_YOUR_MODEL_NAME",
        "OPENAI_API_KEY": "REPLACE_WITH_YOUR_OPENAI_API_KEY",
        "MIDSCENE_MCP_ANDROID_MODE": "true",
        "MCP_SERVER_REQUEST_TIMEOUT": "800000"
      }
    }
  }
}
```
```

--------------------------------

### YAML Script with Full Structure Example

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/automate-with-scripts-in-yaml.mdx

This example demonstrates a complete YAML script structure, including web environment configuration, AI agent settings, and a task flow with AI actions and assertions.

```yaml
web:
  url: https://www.bing.com

# The tasks section defines the series of steps to be executed
tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - sleep: 3000
      - aiAssert: The results show weather information
```

--------------------------------

### Start Chrome Extension Development Server

Source: https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md

Navigate to the chrome-extension directory and start the development server to work on the extension.

```shell
cd apps/chrome-extension
pnpm run dev
```

--------------------------------

### Install huggingface_hub with pip

Source: https://github.com/web-infra-dev/midscene/blob/main/packages/evaluation/README.md

Installs the huggingface_hub library using pip. This is a prerequisite for certain functionalities.

```bash
pip install huggingface_hub
```

--------------------------------

### Prepare and Run Demo Project

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-any-interface.mdx

Clone the demo project, install dependencies, build the project, and run the demo to see Midscene Agent in action with a custom interface.

```bash
git clone https://github.com/web-infra-dev/midscene-example.git
cd midscene-example/custom-interface
npm install
npm run build

# run the demo
npm run demo
```

--------------------------------

### Basic CMake Setup

Source: https://github.com/web-infra-dev/midscene/blob/main/packages/computer/native/rdp/CMakeLists.txt

Sets the minimum CMake version, project name, and C++ standard. Configures C++ standard extensions.

```cmake
cmake_minimum_required(VERSION 3.20)

project(midscene_rdp_helper LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
```

--------------------------------

### Install Android Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-android.mdx

Install the necessary Midscene Android package and dotenv for environment variables.

```bash
npm install @midscene/android dotenv --save-dev
```

--------------------------------

### Install Headless Linux Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/computer-getting-started.mdx

Install necessary dependencies for running desktop automation on a headless Linux server, including Xvfb and ImageMagick.

```bash
sudo apt-get install -y xvfb x11-xserver-utils imagemagick
```

--------------------------------

### Install Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-puppeteer.mdx

Install necessary packages for Puppeteer integration with Midscene.

```bash
npm install @midscene/web puppeteer tsx dotenv --save-dev
```

--------------------------------

### Run Monorepo-Wide Development Server

Source: https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md

Start the root development command which prepares assets and initiates Nx watch builds across packages. Use this for monorepo-wide changes.

```bash
pnpm run dev
```

--------------------------------

### Run Specific App Dev Server

Source: https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md

Start the development server for a single application. Navigate to the app's directory first.

```bash
cd apps/report && pnpm run dev
cd apps/site && pnpm run dev
cd apps/playground && pnpm run dev
cd apps/chrome-extension && pnpm run dev
```

--------------------------------

### Ebay Search Test Case with Midscene AI Actions

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-playwright.mdx

An example test case for searching headphones on eBay using Midscene's AI actions like aiInput, aiTap, aiWaitFor, aiScroll, aiQuery, and aiAssert. Includes setup for viewport, navigation, and network idle wait.

```typescript
import { expect } from '@playwright/test';
import { test } from './fixture';

test.beforeEach(async ({ page }) => {
  page.setViewportSize({ width: 400, height: 905 });
  await page.goto('https://www.ebay.com');
  await page.waitForLoadState('networkidle');
});

test('search headphone on ebay', async ({
  ai,
  aiQuery,
  aiAssert,
  aiInput,
  aiTap,
  aiScroll,
  aiWaitFor,
  aiRightClick,
  recordToReport,
}) => {
  // Use aiInput to enter search keyword
  await aiInput('Headphones', 'search box');

  // Use aiTap to click search button
  await aiTap('search button');

  // Wait for search results to load
  await aiWaitFor('search results list loaded', { timeoutMs: 5000 });

  // Use aiScroll to scroll to bottom
  await aiScroll(
    {
      scrollType: 'untilBottom',
    },
    'search results list',
  );

  // Use aiQuery to get product information
  const items = await aiQuery<Array<{ title: string; price: number }>>(
    'get product titles and prices from search results',
  );

  console.log('headphones in stock', items);
  expect(items?.length).toBeGreaterThan(0);

  // Use aiAssert to verify filter functionality
  await aiAssert('category filter exists on the left side');

  // Use recordToReport to capture the current state
  await recordToReport('Search Results', {
    content: 'Final search results for headphones',
  });
});

```

--------------------------------

### Quick Start with HarmonyAgent

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/harmony-api-reference.mdx

Connect to a HarmonyOS device, launch an app, perform an AI-driven action, and query visible UI elements.

```typescript
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const [first] = await getConnectedDevices();
const device = new HarmonyDevice(first.deviceId, {});
await device.connect();

const agent = new HarmonyAgent(device, {
  aiActionContext: 'This is a HarmonyOS device. If any popup appears, agree to it.',
});

await agent.launch('com.huawei.hmos.settings');
await agent.aiAct('scroll down one screen');
const items = await agent.aiQuery(
  'string[], list all visible setting item names',
);
console.log(items);
```

--------------------------------

### Install Midscene Harmony Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/harmony-getting-started.mdx

Install the necessary Midscene Harmony SDK and dotenv for managing environment variables. Use this command with your preferred package manager.

```bash
npm install @midscene/harmony dotenv --save-dev
```

--------------------------------

### Open New Tab Example

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/web-api-reference.mdx

Demonstrates opening a new tab with a URL and interacting with the agent.

```typescript
import { AgentOverChromeBridge } from '@midscene/web/bridge-mode';

const agent = new AgentOverChromeBridge();
await agent.connectNewTabWithUrl('https://www.bing.com');

await agent.ai('search "AI automation" and summarise first result');
await agent.aiAssert('some search results show up');
await agent.destroy();
```

--------------------------------

### Download ScreenSpot-v2 with npm

Source: https://github.com/web-infra-dev/midscene/blob/main/packages/evaluation/README.md

Downloads the ScreenSpot-v2 model using an npm script. Ensure Node.js and npm are installed.

```bash
npm run download-screenspot-v2
```

--------------------------------

### Install Midscene iOS Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/ios-getting-started.mdx

Install the necessary Midscene iOS SDK and dotenv for managing environment variables.

```bash
npx install @midscene/ios dotenv --save-dev
```

--------------------------------

### Install Midscene Android SDK Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/android-getting-started.mdx

Install the necessary Midscene Android SDK packages and dotenv for environment variable management.

```bash
npm install @midscene/android dotenv --save-dev
```

--------------------------------

### agent.aiInput() Usage Examples

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/api.mdx

Demonstrates how to use the `agent.aiInput()` function with both the recommended and backward-compatible signatures.

```typescript
// Recommended
await agent.aiInput('The search input box', { value: 'Hello World' });
```

```typescript
// Backward compatible (not recommended)
await agent.aiInput('Hello World', 'The search input box');
```

--------------------------------

### Enable pnpm and Install Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md

Commands to enable pnpm via corepack and install project dependencies. This process also sets up symlinks and runs the prepare script for building packages.

```bash
corepack enable
pnpm install
```

--------------------------------

### Install Playwright with Mirror

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-playwright.mdx

Speed up Playwright browser binary downloads by using a mirror. This is useful on slower networks.

```bash
PLAYWRIGHT_DOWNLOAD_HOST="https://npmmirror.com/mirrors/playwright" npx playwright install
```

--------------------------------

### Initialize AndroidDevice with default options

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/android-api-reference.mdx

Create a new AndroidDevice instance using a device ID. This is the basic setup for controlling an Android device.

```typescript
const device = new AndroidDevice(deviceId, {
  // device options...
});
```

--------------------------------

### Install Midscene Skills

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/skills.mdx

Install the general Midscene skills package. You can also add specific configurations for agents like Claude Code or OpenClaw.

```bash
npx skills add web-infra-dev/midscene-skills
```

```bash
npx skills add web-infra-dev/midscene-skills -a claude-code
```

```bash
npx skills add web-infra-dev/midscene-skills -a openclaw
```

--------------------------------

### Run First Midscene Script

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/computer-getting-started.mdx

Execute your first desktop automation script using tsx. Ensure you have Node.js and tsx installed.

```bash
npx tsx example.ts
```

--------------------------------

### Use Skills in AI Chat Assistant

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/skills.mdx

Example command to interact with skills within an AI chat assistant. This specific example instructs the assistant to open a photo app and describe its first photo.

```markdown
Open photo app, see what is the first photo in the album.
```

--------------------------------

### Install and Load Dotenv for JavaScript SDK

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/model-common-config.mdx

Install the dotenv package and import it in your script to load variables from a .env file into Node.js process.env. The .env file should not use the 'export' prefix.

```bash
npm install dotenv --save
```

```dotenv
MIDSCENE_MODEL_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

```typescript
import 'dotenv/config';
```

--------------------------------

### Launch Midscene Android Playground

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/android-getting-started.mdx

Start the interactive Playground for validating device connection and observing AI-driven actions without writing code.

```bash
npx --yes @midscene/android-playground
```

--------------------------------

### Launch App or Navigate URL on Android

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/mcp-android.mdx

Use the `midscene_android_launch` tool to start an application or open a specific URL on the connected Android device.

```text
Parameters:
  - uri: Package name, activity name, or URL to launch
```

--------------------------------

### Full Vitest Example with ReportMergingTool

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/api.mdx

A comprehensive example demonstrating the integration of ReportMergingTool within a Vitest suite for Android automation. It includes setup, report appending, and final merging.

```typescript
import { describe, it, beforeEach, afterEach, afterAll } from 'vitest';
import { AndroidAgent, AndroidDevice } from '@midscene/android';
import { ReportMergingTool } from '@midscene/core/report';

describe('Android settings automation', () => {
  let device: AndroidDevice;
  let agent: AndroidAgent;
  let startTime: number;
  const reportMergingTool = new ReportMergingTool();

  beforeEach((ctx) => {
    startTime = performance.now();
    agent = new AndroidAgent(device, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    let workflowStatus = 'passed';
    if (ctx.task.result?.state === 'pass') {
      workflowStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      workflowStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0]?.message.includes('timed out')) {
      workflowStatus = 'timedOut';
    } else {
      workflowStatus = 'failed';
    }

    // Add the report to the merge list
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: ctx.task.name,
        testTitle: ctx.task.name,
        testDescription: 'Automation workflow description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: workflowStatus,
      },
    });
  });

  afterAll(() => {
    // Merge all automation reports
    reportMergingTool.mergeReports('android-settings-automation-report');
  });

  it('toggle WLAN', async () => {
    await agent.aiAct('Find and open WLAN settings');
    await agent.aiAct('Toggle WLAN once');
  });

  it('toggle Bluetooth', async () => {
    await agent.aiAct('Find and open Bluetooth settings');
    await agent.aiAct('Toggle Bluetooth once');
  });
});
```

--------------------------------

### Example GitHub Sign-up Prompt

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/model-strategy.mdx

This prompt requires a model to understand per-field rules, identify controls, operate multi-step selectors, paginate, trigger validation, and locate elements. It's used to illustrate the need for advanced model strategies.

```text
Fill out the GitHub sign-up form. Ensure no fields are missing and pick “United States” as the region.

Make sure every field passes validation.

Only fill the form; do not submit a real registration request.

Return the actual field values you filled.
```

--------------------------------

### Configure Qwen3-VL

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/model-common-config.mdx

Set environment variables to use Alibaba Cloud's Qwen3-VL models. The base URL is for the compatible mode. Midscene disables model-native thinking by default.

```bash
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen3-vl-plus"
MIDSCENE_MODEL_FAMILY="qwen3-vl"
```

--------------------------------

### Quick Start with PuppeteerAgent

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/web-api-reference.mdx

Demonstrates a basic workflow using PuppeteerAgent: launching Puppeteer, navigating, performing an AI action, querying data, asserting a condition, and closing the browser.

```typescript
import puppeteer from 'puppeteer';
import { PuppeteerAgent } from '@midscene/web/puppeteer';

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PuppeteerAgent(page, {
  actionContext: 'When a cookie dialog appears, accept it.',
});

await agent.aiAct('search "Noise cancelling headphones" and open first result');
const items = await agent.aiQuery(
  '{itemTitle: string, price: number}[], list two products with price',
);
console.log(items);

await agent.aiAssert('there is a category filter on the left sidebar');
await browser.close();
```

--------------------------------

### Verify HDC Installation

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/harmony-getting-started.mdx

Run this command to check if HDC is installed on your system. A version number indicates a successful installation.

```bash
hdc version
```

--------------------------------

### Verify ADB Installation

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/common/prepare-android.mdx

Run this command to check if the Android Debug Bridge (adb) is installed correctly on your system. Successful output confirms the installation.

```bash
adb --version
```

--------------------------------

### YAML Script with Agent and Environment Configuration

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/automate-with-scripts-in-yaml.mdx

This comprehensive example shows a YAML script with detailed agent settings, and configurations for both iOS and Android environments, alongside a task flow.

```yaml
# agent configuration, applies to all environments
agent:
  testId: "checkout-test"
  groupName: "E2E Test Suite"
  groupDescription: "Complete checkout flow testing"
  generateReport: true
  autoPrintReportMsg: false
  reportFileName: "checkout-report"
  replanningCycleLimit: 30
  aiActContext: "If any popup appears, click agree. If login page appears, skip it."
  cache:
    id: "checkout-cache"
    strategy: "read-write"

# iOS environment configuration
ios:
  launch: https://www.bing.com
  wdaPort: 8100

# Or Android environment configuration
android:
  deviceId: s4ey59
  launch: https://www.bing.com

tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - aiAssert: The results show weather information
```

--------------------------------

### Install Langfuse and OpenTelemetry Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/model-config.mdx

Install the necessary npm packages for Langfuse OpenAI integration and OpenTelemetry.

```bash
npm install @langfuse/openai @langfuse/otel @opentelemetry/sdk-node
```

--------------------------------

### Install LangSmith Dependency

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/model-config.mdx

Install the necessary 'langsmith' npm package to enable auto-integration with LangSmith for LLM debugging.

```bash
npm install langsmith
```

--------------------------------

### Build All Packages

Source: https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md

Run the pnpm build script to build all packages in the monorepo.

```bash
pnpm run build
```

--------------------------------

### Install Specific Playwright Browser

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-playwright.mdx

Install only the Chromium browser and its dependencies for Playwright. This reduces download time and disk space.

```bash
npx playwright install --with-deps chromium
```

--------------------------------

### Install Playwright and Midscene Dependencies

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-playwright.mdx

Install the necessary packages for Playwright and Midscene integration using npm, yarn, or pnpm.

```bash
npm install @midscene/web playwright @playwright/test tsx --save-dev
```

--------------------------------

### Install Dependencies for Remote Puppeteer

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-puppeteer.mdx

Install Puppeteer and Midscene web packages as development dependencies for remote browser integration.

```bash
npm install puppeteer @midscene/web --save-dev

```

--------------------------------

### Launch App or URL on iOS

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/automate-with-scripts-in-yaml.mdx

Use the `launch` action to open an iOS application by its bundle ID or navigate to a URL.

```yaml
ios:
  wdaPort: 8100

tasks:
  - name: Launch Settings app
    flow:
      - launch: com.apple.Preferences

  - name: Open webpage
    flow:
      - launch: https://www.example.com
```

--------------------------------

### PuppeteerAgent Remote Browser Connection Example

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/web-api-reference.mdx

Example of connecting PuppeteerAgent to a remote browser instance using a WebSocket endpoint.

```APIDOC
## Connect to a remote Puppeteer browser

### Description
This example demonstrates how to connect to a remotely running Puppeteer browser instance and use the `PuppeteerAgent` for automation.

### Usage
```ts
import puppeteer from 'puppeteer';
import { PuppeteerAgent } from '@midscene/web/puppeteer';

const browser = await puppeteer.connect({
  browserWSEndpoint: process.env.REMOTE_CDP_URL!,
});

const [page = await browser.newPage()] = await browser.pages();
const agent = new PuppeteerAgent(page, {
  waitForNetworkIdleTimeout: 0,
});

await agent.aiAct('open https://example.com and click the login button');
await agent.destroy();
await browser.disconnect();
```
```

--------------------------------

### Instantiate IOSDevice

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/ios-api-reference.mdx

Create an instance of IOSDevice with default options.

```typescript
const device = new IOSDevice({
  // device options...
});
```

--------------------------------

### Install Midscene CLI Per Project

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/yaml-script-runner.mdx

Command to install the Midscene CLI as a development dependency within a specific project using npm.

```bash
npm i @midscene/cli --save-dev
```

--------------------------------

### Configure Qwen3.x Series Model

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/model-common-config.mdx

Set environment variables for Qwen3.x models, including base URL, API key, and model name/family. Optionally, enable model-native thinking and control reasoning budget.

```bash
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen3.5-plus" # For Qwen3.6, use "qwen3.6-plus"
MIDSCENE_MODEL_FAMILY="qwen3.5" # For Qwen3.6, use "qwen3.6"
```

```bash
MIDSCENE_MODEL_REASONING_ENABLED="true"
```

```bash
MIDSCENE_MODEL_REASONING_BUDGET="500"
```

--------------------------------

### Puppeteer Installation: Skip Browser Download

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-puppeteer.mdx

Use `PUPPETEER_SKIP_DOWNLOAD=true` to skip browser download during npm install, then manually download the browser binary.

```bash
# Skip browser download first so dependency installation can finish
PUPPETEER_SKIP_DOWNLOAD=true npm install

# Then download Chrome from a mirror
# This example uses https://registry.npmmirror.com
# Different mirror providers may use different base-url rules
npx puppeteer browsers install chrome --base-url="https://registry.npmmirror.com/-/binary/chrome-for-testing"
```

--------------------------------

### Open Application and Navigate

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/computer-api-reference.mdx

Opens a text editor application (TextEdit on macOS, Notepad on Windows) and types content into it. It then saves the file using platform-specific shortcuts.

```typescript
import { agentForComputer } from '@midscene/computer';

const agent = await agentForComputer();

// Open application
if (process.platform === 'darwin') {
  await agent.aiAct('press Cmd+Space');
  await agent.aiAct('type "TextEdit" and press Enter');
} else {
  await agent.aiAct('press Windows key');
  await agent.aiAct('type "Notepad" and press Enter');
}

await agent.aiWaitFor('text editor window is visible');

// Type content
await agent.aiAct('type "Hello, Midscene!"');

// Save file
if (process.platform === 'darwin') {
  await agent.aiAct('press Cmd+S');
} else {
  await agent.aiAct('press Ctrl+S');
}
```

--------------------------------

### Launch and Navigate Apps on HarmonyOS

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/harmony-api-reference.mdx

Demonstrates launching specific applications and using basic navigation actions like back and home.

```typescript
await agent.launch('com.huawei.hmos.settings'); // Open Settings
await agent.launch('com.huawei.hmos.camera');    // Open Camera
await agent.back();
await agent.home();
```

--------------------------------

### Launch App or URL on Android

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/automate-with-scripts-in-yaml.mdx

Use the `launch` action to open an Android application by its package name or navigate to a URL.

```yaml
android:
  deviceId: 'test-device'

tasks:
  - name: Launch Settings app
    flow:
      - launch: com.android.settings

  - name: Open webpage
    flow:
      - launch: https://www.example.com
```

--------------------------------

### Check Node.js Version

Source: https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md

Use this command to check your currently installed Node.js version.

```bash
node -v
```

--------------------------------

### Basic Android Automation Script with Midscene SDK

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/android-getting-started.mdx

A TypeScript example demonstrating how to open a browser, search on eBay, and assert results using the Midscene Android SDK. Ensure you have a .env file for API keys if needed.

```typescript
import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new AndroidDevice(devices[0].udid);

    const agent = new AndroidAgent(device, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await device.connect();

    await agent.aiAct('open browser and navigate to ebay.com');
    await sleep(5000);
    await agent.aiAct('type "Headphones" in search box, hit Enter');
    await agent.aiWaitFor('There is at least one headphone product');

    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    await agent.aiAssert('There is a category filter on the left');
  })(),
);

```

--------------------------------

### List and Connect to Specific Displays

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/computer-getting-started.mdx

This code demonstrates how to list all available displays on the system and then connect to a specific one using its ID. This is useful when dealing with multi-monitor setups.

```typescript
import { ComputerDevice, agentForComputer } from '@midscene/computer';

// List all displays
const displays = await ComputerDevice.listDisplays();
console.log('Available displays:', displays);

// Connect to a specific display
const agent = await agentForComputer({
  displayId: displays[0].id,
});
```

--------------------------------

### agent.aiScroll Backward Compatible Usage Example

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/api.mdx

Shows how to use the backward-compatible signature of agent.aiScroll for scrolling.

```typescript
// Backward compatible (not recommended)
await agent.aiScroll(
  { scrollType: 'singleAction', direction: 'up', distance: 100 },
  'The form panel',
);
```

--------------------------------

### Launch URL or App

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/ios-api-reference.mdx

Launches a web URL, native application bundle, or custom scheme. Supports app names mapped to Bundle IDs.

```typescript
function launch(target: string): Promise<void>;
```

```typescript
await agent.launch('https://www.apple.com');
```

```typescript
await agent.launch('com.apple.Preferences');
```

```typescript
await agent.launch('myapp://profile/user/123');
```

```typescript
await agent.launch('tel:+1234567890');
```

--------------------------------

### agent.aiScroll Recommended Usage Example

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/api.mdx

Demonstrates how to use the recommended signature of agent.aiScroll to scroll a specific element.

```typescript
// Recommended
await agent.aiScroll('The form panel', {
  scrollType: 'singleAction',
  direction: 'up',
  distance: 100,
});
```

--------------------------------

### Attach to Current Tab Example

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/web-api-reference.mdx

Shows how to attach the agent to the current tab with specific configuration options.

```typescript
import { AgentOverChromeBridge } from '@midscene/web/bridge-mode';

const agent = new AgentOverChromeBridge({
  allowRemoteAccess: false,
  closeNewTabsAfterDisconnect: true,
});

await agent.connectCurrentTab({ forceSameTabNavigation: true });
await agent.aiAct('open Gmail and report how many unread emails are visible');
await agent.destroy();
```

--------------------------------

### connectNewTabWithUrl

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/web-api-reference.mdx

Opens a new desktop tab with the specified URL and then connects the agent to it.

```APIDOC
## connectNewTabWithUrl(url, options?)

### Description
Opens a new desktop tab with the specified URL and then connects the agent to it.

### Method
`connectNewTabWithUrl(url: string, options?: { forceSameTabNavigation?: boolean }): Promise<void>`

### Parameters
#### Path Parameters
- `url` (string) - Required - The URL to open in the new desktop tab.
#### Options
- `options.forceSameTabNavigation` (boolean) - Optional. Same as in `connectCurrentTab`.

### Response
- Resolves when the new tab is opened and the bridge is connected.
```

--------------------------------

### Workflow Style Automation with aiQuery and aiTap

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/introduction.mdx

Implement workflow-style automation by splitting complex logic into steps. This example demonstrates querying a list of records, checking each for a "completed" status, and tapping non-completed records. Requires the `agent` object to be initialized.

```javascript
const recordList = await agent.aiQuery('string[], the record list')
for (const record of recordList) {
  const hasCompleted = await agent.aiBoolean(`check if the record ${record}" contains the text "completed"`)
  if (!hasCompleted) {
    await agent.aiTap(record)
  }
}
```

--------------------------------

### Prompt with Images for Assert Action

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/automate-with-scripts-in-yaml.mdx

For insight steps like `aiAssert`, `prompt` and `images` can be set directly. `convertHttpImage2Base64: true` is useful for remote images that need to be sent as Base64.

```yaml
tasks:
  - name: Verify branding
    flow:
      - aiAssert:
          prompt: Check whether the image appears on the page.
          images:
            - name: target logo
              url: https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png
          convertHttpImage2Base64: true
```

--------------------------------

### Launch Webpage or Native Page on Android

Source: https://github.com/web-infra-dev/midscene/blob/main/apps/site/docs/en/integrate-with-android.mdx

Use `agent.launch()` to open a specified URI. This can be a web URL or a native Android activity. Ensure the AndroidAgent and AndroidDevice are initialized.

```typescript
import { AndroidAgent, AndroidDevice } from '@midscene/android';

const device = new AndroidDevice('s4ey59');
const agent = new AndroidAgent(device);

await agent.launch('https://www.ebay.com'); // open a webpage
await agent.launch('com.android.settings'); // open a native page
await agent.launch('com.android.settings/.Settings'); // open a native page
```