Midscene.js

https://midscenejs.com/
Admin
Midscene.js is an open-source AI operator for joyful automation across web and mobile apps, enabling...
Tokens:145,540
Snippets:2,050
Trust Score:7.4
Update:2 months ago
Show doc for...
Tokens:
Raw
### Verify ADB version output

Source: https://midscenejs.com/llms-full.txt

Example output confirming a successful ADB installation.

```log
Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64)
```

--------------------------------

### Usage example for playgroundForAgent

Source: https://midscenejs.com/integrate-with-any-interface

A complete example showing how to initialize an agent, launch the playground, and shut it down.

```typescript
import 'dotenv/config';
import { playgroundForAgent } from '@midscene/playground';
import { SampleDevice } from './sample-device';
import { Agent } from '@midscene/core/agent';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// Create device and agent instances
const device = new SampleDevice();
const agent = new Agent(device);

// Launch playground
const result = await playgroundForAgent(agent).launch({
  port: 5800,
  openBrowser: true,
  verbose: true
});

console.log(`Playground started: http://${result.host}:${result.port}`);

// Close playground when needed
await sleep(10 * 60 * 1000); // Wait 10 minutes
await result.close();
console.log('Playground closed!');
```

--------------------------------

### aiRightClick usage examples

Source: https://midscenejs.com/reference

Examples demonstrating standard right-click and right-click with deepLocate enabled.

```javascript
await agent.aiRightClick('The file name at the top of the page');

// Use deepLocate feature to precisely locate the element
await agent.aiRightClick('The file name at the top of the page', {
  deepLocate: true,
});
```

--------------------------------

### launch(options)

Source: https://midscenejs.com/integrate-with-any-interface

Starts the playground server with the specified configuration.

```APIDOC
## launch(options)

### Description
Launches the playground server instance based on the provided configuration options.

### Parameters
- **port** (number) - Optional - Port to start the playground server on (default: 5800).
- **openBrowser** (boolean) - Optional - Whether to automatically open the playground in browser (default: true).
- **browserCommand** (string) - Optional - Custom browser command to open playground.
- **verbose** (boolean) - Optional - Whether to show server logs (default: true).
- **id** (string) - Optional - Unique identifier for the playground server instance.

### Return Value
Returns a `LaunchPlaygroundResult` object containing:
- **server** (PlaygroundServer) - The playground server instance.
- **port** (number) - The server port.
- **host** (string) - The server host.
- **close** (Function) - A function to close the playground.
```

--------------------------------

### Install Midscene and Puppeteer

Source: https://midscenejs.com/integrate-with-puppeteer

Installation commands for various package managers to set up the required dependencies.

```bash
npm install puppeteer @midscene/web --save-dev
```

```bash
yarn add puppeteer @midscene/web --save-dev
```

```bash
pnpm add puppeteer @midscene/web --save-dev
```

```bash
bun add puppeteer @midscene/web --save-dev
```

```bash
deno add npm:puppeteer npm:@midscene/web --save-dev
```

--------------------------------

### Record and query UI observations

Source: https://midscenejs.com/reference

Example workflow for starting an observation, performing actions, stopping the recording, and querying the captured frames.

```typescript
const observer = await agent.startObserving();
await agent.aiAct('submit the form');
const observation = await observer.stop();
await observation.aiAssert('a success toast appeared during the process');
const toastCount = await observation.aiNumber(
  'how many success toasts appeared?',
);
await observation.dispose();
```

--------------------------------

### Install Dependencies

Source: https://midscenejs.com/integrate-with-puppeteer

Install Midscene, Puppeteer, and supporting tools using your preferred package manager.

```bash
npm install @midscene/web puppeteer tsx dotenv --save-dev
```

```bash
yarn add @midscene/web puppeteer tsx dotenv --save-dev
```

```bash
pnpm add @midscene/web puppeteer tsx dotenv --save-dev
```

```bash
bun add @midscene/web puppeteer tsx dotenv --save-dev
```

```bash
deno add npm:@midscene/web npm:puppeteer npm:tsx npm:dotenv --save-dev
```

--------------------------------

### Setup Demo Project

Source: https://midscenejs.com/integrate-with-any-interface

Commands to clone and initialize the custom interface demo project.

```bash
# prepare the environment
git clone https://github.com/web-infra-dev/midscene-example.git
cd midscene-example/custom-interface
npm install
npm run build

# run the demo
npm run demo
```

--------------------------------

### Install specific browser

Source: https://midscenejs.com/integrate-with-playwright

Install only the Chromium browser to reduce download size and time.

```bash
npx playwright install --with-deps chromium
```

--------------------------------

### Run setup script with shared browser context

Source: https://midscenejs.com/yaml-script-runner

Configure a prerequisite setup script to run before parallel execution, sharing the browser context across all scripts.

```yaml
setup: ./scripts/login.yaml

files:
  - ./scripts/search.yaml
  - ./scripts/report.yaml
  - ./scripts/settings.yaml

shareBrowserContext: true
concurrent: 3
```

--------------------------------

### Install Midscene CLI

Source: https://midscenejs.com/yaml-script-runner

Commands to install the Midscene CLI globally or as a project dependency.

```bash
npm i -g @midscene/cli
```

```bash
npm i @midscene/cli --save-dev
```

--------------------------------

### Verify ANDROID_HOME output

Source: https://midscenejs.com/llms-full.txt

Example output showing a configured Android SDK path.

```log
/Users/your_username/Library/Android/sdk
```

--------------------------------

### Midscene Core API Usage Examples

Source: https://midscenejs.com/llms-full.txt

Examples of natural language prompts for Act, Query, Assert, and Tap operations.

```text
Type “Midscene” in the search box, run the search, and open the first result
```

```text
Fill out the registration form and make sure every field passes validation
```

```text
Extract the user ID from the page and return JSON data in the { id: string } structure
```

```text
There is a login button on the page, with a user agreement link below it
```

```text
Click the login button
```

--------------------------------

### aiKeyboardPress Usage Examples

Source: https://midscenejs.com/reference

Demonstrates how to use the recommended and legacy signatures to perform keyboard actions.

```javascript
// Recommended
await agent.aiKeyboardPress('The search input box', { keyName: 'Enter' });
await agent.aiKeyboardPress('The search input box', { keyName: 'Control+A' });

// Backward compatible (not recommended)
await agent.aiKeyboardPress('Enter', 'The search input box');
```

--------------------------------

### GitHub Sign-up Prompt Example

Source: https://midscenejs.com/model-strategy.html

A complex prompt requiring multi-step interaction, validation, and data extraction.

```text
Fill out the GitHub sign-up form. Ensure no fields are missing and pick “United States” as the region.

Make sure every field passes validation.

Only fill the form; do not submit a real registration request.

Return the actual field values you filled.
```

--------------------------------

### Install @midscene/computer dependencies

Source: https://midscenejs.com/platforms/desktop

Install the required computer automation package using your preferred package manager.

```bash
npm install @midscene/computer
```

```bash
yarn add @midscene/computer
```

```bash
pnpm add @midscene/computer
```

```bash
bun add @midscene/computer
```

```bash
deno add npm:@midscene/computer
```

--------------------------------

### Launch HarmonyOS apps

Source: https://midscenejs.com/reference/index.html

Examples of launching specific apps using their bundle names.

```typescript
await agent.launch('com.huawei.hmos.settings'); // Open Settings
await agent.launch('com.huawei.hmos.camera');    // Open Camera
```

--------------------------------

### aiDoubleClick usage examples

Source: https://midscenejs.com/reference

Examples demonstrating standard double-click and double-click with deepLocate enabled.

```javascript
await agent.aiDoubleClick('The file name at the top of the page');

// Use deepLocate feature to precisely locate the element
await agent.aiDoubleClick('The file name at the top of the page', {
  deepLocate: true,
});
```

--------------------------------

### Install Linux Desktop Dependencies

Source: https://midscenejs.com/platforms/desktop

Required system packages for screenshot and display functionality on Linux environments.

```bash
# Install dependencies
sudo apt-get install -y xvfb x11-xserver-utils imagemagick
```

--------------------------------

### Define Web and Browser Targets

Source: https://midscenejs.com/llms-full.txt

Examples of configuring page-level, browser-level, and compatibility-mode targets.

```yaml
page:
  url: https://example.com
```

```yaml
browser:
  url: https://example.com
  autoFollowNewPage: true
```

```yaml
web:
  mode: browser
  url: https://example.com
  autoFollowNewPage: true
```

--------------------------------

### Assert Tab Prompt

Source: https://midscenejs.com/quick-experience

Example prompt for verifying interface conditions.

```text
There is a login button on the page, with a user agreement link below it
```

--------------------------------

### Install Linux Headless Dependencies

Source: https://midscenejs.com/platforms/desktop

Installs required packages for running Midscene in a headless Linux environment.

```bash
sudo apt-get install -y xvfb x11-xserver-utils imagemagick
```

--------------------------------

### Install Midscene Dependencies

Source: https://midscenejs.com/platforms/harmonyos

Commands to install the required HarmonyOS automation package and dotenv for environment management.

```bash
npm install @midscene/harmony dotenv --save-dev
```

```bash
yarn add @midscene/harmony dotenv --save-dev
```

```bash
pnpm add @midscene/harmony dotenv --save-dev
```

```bash
bun add @midscene/harmony dotenv --save-dev
```

```bash
deno add npm:@midscene/harmony npm:dotenv --save-dev
```

--------------------------------

### aiPinch Usage Examples

Source: https://midscenejs.com/reference

Demonstrates various ways to trigger pinch gestures for zooming in or out.

```typescript
// Zoom in on the map (spread fingers apart)
await agent.aiPinch('the map area', { direction: 'out', distance: 200 });

// Zoom out at screen center (pinch fingers together)
await agent.aiPinch(undefined, { direction: 'in' });

// Zoom in with custom duration
await agent.aiPinch('the image', { direction: 'out', distance: 300, duration: 1000 });
```

--------------------------------

### aiTap usage examples

Source: https://midscenejs.com/reference

Examples for tapping elements, including deep location and file upload handling.

```typescript
await agent.aiTap('The login button at the top of the page');

// Use deepLocate feature to precisely locate the element
await agent.aiTap('The login button at the top of the page', {
  deepLocate: true,
});

// File upload: tap the upload button and select files
await agent.aiTap('Choose file button', { fileChooserAccept: ['./document.pdf'] });
await agent.aiTap('Upload images', { fileChooserAccept: ['./image1.jpg', './image2.png'] });
```

--------------------------------

### Launch HarmonyOS Apps

Source: https://midscenejs.com/automate-with-scripts-in-yaml.html

Starts a HarmonyOS application using its bundle name.

```yaml
harmony:
  deviceId: 'test-device'

tasks:
  - name: Launch app
    flow:
      - launch: com.example.app
```

--------------------------------

### Wait for conditions with aiWaitFor

Source: https://midscenejs.com/reference

Examples demonstrating basic usage and custom configuration for waiting on UI conditions.

```typescript
// Basic usage
await agent.aiWaitFor(
  'There is at least one headphone information displayed on the interface',
);

// Using custom options
await agent.aiWaitFor('The shopping cart icon shows a quantity of 2', {
  timeoutMs: 30000, // Wait for 30 seconds
  checkIntervalMs: 5000, // Check every 5 seconds
});
```

--------------------------------

### aiInput Usage Examples

Source: https://midscenejs.com/reference

Demonstrates both the recommended and legacy usage patterns for entering text into an input field.

```typescript
// Recommended
await agent.aiInput('The search input box', { value: 'Hello World' });

// Backward compatible (not recommended)
await agent.aiInput('Hello World', 'The search input box');
```

--------------------------------

### Android Device Connection Output

Source: https://midscenejs.com/llms-full.txt

Example output indicating a successful Android device connection.

```log
List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3
```

--------------------------------

### Using reference images in AI methods

Source: https://midscenejs.com/reference

Examples of passing images to aiTap, aiAssert, and aiAct to guide AI decision-making.

```javascript
await agent.aiTap({
  prompt: 'The specific logo',
  images: [
    {
      name: 'The specific logo',
      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
    },
  ],
});
```

```javascript
await agent.aiAssert({
  prompt: 'Whether there is a specific logo on the page.',
  images: [
    {
      name: 'The specific logo',
      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
    },
  ],
});
```

```javascript
await agent.aiAct({
  prompt: 'Tap the icon that matches the reference logo',
  images: [
    {
      name: 'The specific logo',
      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
    },
  ],
});
```

--------------------------------

### Complete Agent and Environment Configuration Example

Source: https://midscenejs.com/automate-with-scripts-in-yaml.html

Demonstrates configuring an AI agent alongside specific environment settings for iOS or Android.

```yaml
# agent configuration, applies to all environments
agent:
  testId: "checkout-test"
  groupName: "E2E Test Suite"
  groupDescription: "Complete checkout flow testing"
  generateReport: true
  autoPrintReportMsg: false
  reportFileName: "checkout-report"
  replanningCycleLimit: 30
  aiActContext: "If any popup appears, click agree. If login page appears, skip it."
  cache:
    id: "checkout-cache"
    strategy: "read-write"

# iOS environment configuration
ios:
  launch: https://www.bing.com
  wdaPort: 8100

# Or Android environment configuration
android:
  deviceId: s4ey59
  launch: https://www.bing.com

tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - aiAssert: The results show weather information
```

--------------------------------

### Bypass Puppeteer browser download during installation

Source: https://midscenejs.com/integrate-with-puppeteer

Use environment variables to skip the automatic browser download if installation hangs, then install the browser manually from a mirror.

```bash
# Skip browser download first so dependency installation can finish
PUPPETEER_SKIP_DOWNLOAD=true npm install

# Then download Chrome from a mirror
# This example uses https://registry.npmmirror.com
# Different mirror providers may use different base-url rules
npx puppeteer browsers install chrome --base-url="https://registry.npmmirror.com/-/binary/chrome-for-testing"
```

--------------------------------

### aiHover usage example

Source: https://midscenejs.com/reference

Example for hovering over a specific element on the page.

```typescript
await agent.aiHover('The version number of the current page');
```

--------------------------------

### Install Langfuse Dependencies

Source: https://midscenejs.com/model-config

Install the necessary packages for Langfuse and OpenTelemetry support.

```bash
npm install @langfuse/openai @langfuse/otel @opentelemetry/sdk-node
```

--------------------------------

### Install LangSmith Dependency

Source: https://midscenejs.com/model-config

Install the required package to enable LangSmith integration.

```bash
npm install langsmith
```

--------------------------------

### Write a HarmonyOS Automation Script

Source: https://midscenejs.com/platforms/harmonyos

Example script demonstrating device connection, app launching, scrolling, and AI-based querying.

```typescript
import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new HarmonyDevice(devices[0].deviceId, {});

    const agent = new HarmonyAgent(device, {
      aiActionContext:
        'This is a HarmonyOS device. The system language is Chinese. If any popup appears, dismiss or agree to it.',
    });
    await device.connect();

    // Open Settings app
    await agent.launch('com.huawei.hmos.settings');
    await sleep(2000);

    // Scroll down
    await agent.aiAct('scroll down one screen');

    // Query page content
    const items = await agent.aiQuery(
      'string[], list all visible setting item names',
    );
    console.log('Settings items', items);

    // Assert
    await agent.aiAssert('There is a settings item list on the page');
  })(),
);
```

--------------------------------

### aiLongPress usage examples

Source: https://midscenejs.com/reference

Examples demonstrating standard long-press and long-press with a custom duration.

```javascript
// Long-press an article to open the context menu
await agent.aiLongPress('the first article on the homepage');

// Long-press with a custom duration
await agent.aiLongPress('the message bubble', { duration: 2000 });
```

--------------------------------

### startObserving()

Source: https://midscenejs.com/reference

Starts recording the screen to capture transient UI elements like toasts or banners. Returns a UIObserver to manage the recording session.

```APIDOC
## startObserving(options)

### Description
Records the screen continuously to allow querying or asserting against recorded frames. Use this to capture transient UI states.

### Parameters
- **options** (object) - Optional
  - **intervalMs** (number) - Sampling interval (default 1000ms, min 200ms)
  - **maxFrames** (number) - Maximum number of retained frames (default 30)
  - **watchdogMs** (number) - Maximum recording time (default 300,000ms; 0 disables limit)

### Returns
- **Promise<UIObserver>** - A promise that resolves to a UIObserver instance.
```

--------------------------------

### Launch HarmonyOS Playground

Source: https://midscenejs.com/platforms/harmonyos

Start the interactive Playground CLI to validate connections and test automation capabilities.

```bash
npx --yes @midscene/harmony-playground
```

--------------------------------

### Verify HDC Installation

Source: https://midscenejs.com/platforms/harmonyos

Check if the HarmonyOS Device Connector is correctly installed and accessible in the system.

```bash
hdc version
```

--------------------------------

### Install Midscene Android Dependencies

Source: https://midscenejs.com/platforms/android

Install the required packages using your preferred package manager.

```bash
npm install @midscene/android dotenv --save-dev
```

```bash
yarn add @midscene/android dotenv --save-dev
```

```bash
pnpm add @midscene/android dotenv --save-dev
```

```bash
bun add @midscene/android dotenv --save-dev
```

```bash
deno add npm:@midscene/android npm:dotenv --save-dev
```

--------------------------------

### Verify adb installation

Source: https://midscenejs.com/platforms/android

Check if the Android Debug Bridge is correctly installed and accessible in your terminal.

```bash
adb --version
```

--------------------------------

### Initialize IOSDevice

Source: https://midscenejs.com/reference

Create a new instance of IOSDevice with configuration options.

```typescript
const device = new IOSDevice({
  // device options...
});
```

--------------------------------

### Performing UI Actions with aiAct and ai

Source: https://midscenejs.com/reference

Examples demonstrating basic usage, shorthand syntax, timeout handling with AbortSignal, and enabling deepThink for complex tasks.

```javascript
// Basic usage
await agent.aiAct(
  'Type "JavaScript" into the search box, then click the search button',
);

// Using the shorthand .ai form
await agent.ai(
  'Click the login button at the top of the page, then enter "test@example.com" in the username field',
);


// Using abortSignal to set a timeout
const controller = new AbortController();
setTimeout(() => controller.abort('timeout'), 30000); // 30s timeout
await agent.aiAct('Fill in the form and submit', {
  abortSignal: controller.signal,
});

// For complex tasks, you can enable the deepThink parameter
await agent.aiAct('Complete the GitHub account registration form. The region must be set to "Canada". Make sure no fields on the form are missed and all form fields pass validation. Just fill in the form fields without actually submitting the registration. Finally, return the actual content filled in the form fields', { deepThink: true });
```

--------------------------------

### Install Midscene iOS Dependencies

Source: https://midscenejs.com/platforms/ios

Install the @midscene/ios and dotenv packages using your preferred package manager.

```bash
npm install @midscene/ios dotenv --save-dev
```

```bash
yarn add @midscene/ios dotenv --save-dev
```

```bash
pnpm add @midscene/ios dotenv --save-dev
```

```bash
bun add @midscene/ios dotenv --save-dev
```

```bash
deno add npm:@midscene/ios npm:dotenv --save-dev
```

--------------------------------

### Write an iOS Automation Script

Source: https://midscenejs.com/platforms/ios

Example script to launch Safari, search eBay, and perform AI-driven assertions and data extraction.

```typescript
import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  IOSAgent,
  IOSDevice,
  agentFromWebDriverAgent,
} from '@midscene/ios';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    // Method 1: Create device and agent directly
    const page = new IOSDevice({
      wdaPort: 8100,
      wdaHost: 'localhost',
    });

    // 👀 Initialize Midscene agent
    const agent = new IOSAgent(page, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.',
    });
    await page.connect();

    // Method 2: Or use convenience function (recommended)
    // const agent = await agentFromWebDriverAgent({
    //   wdaPort: 8100,
    //   wdaHost: 'localhost',
    //   aiActionContext: 'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.',
    // });

    // 👀 Directly open ebay.com webpage (recommended approach)
    await page.launch('https://ebay.com');
    await sleep(3000);

    // 👀 Enter keywords and perform search
    await agent.aiAct('Search for "Headphones"');

    // 👀 Wait for loading to complete
    await agent.aiWaitFor('At least one headphone product is displayed on the page');
    // Or you can use a simple sleep:
    // await sleep(5000);

    // 👀 Understand page content and extract data
    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find product titles and prices in the list',
    );
    console.log('Headphone product information', items);

    // 👀 Use AI assertion
    await agent.aiAssert('Multiple headphone products are displayed on the interface');

    await page.destroy();
  })(),
);
```

--------------------------------

### Execute using a configuration file

Source: https://midscenejs.com/llms-full.txt

Run the Midscene CLI using a previously defined YAML configuration file.

```bash
midscene --config ./config.yaml
```

--------------------------------

### Install Midscene Dependencies

Source: https://midscenejs.com/quick-start

Install the required packages for Midscene and Playwright using your preferred package manager.

```bash
npm install @midscene/web playwright tsx --save-dev
```

```bash
yarn add @midscene/web playwright tsx --save-dev
```

```bash
pnpm add @midscene/web playwright tsx --save-dev
```

```bash
bun add @midscene/web playwright tsx --save-dev
```

```bash
deno add npm:@midscene/web npm:playwright npm:tsx --save-dev
```

--------------------------------

### Initialize Agent with Custom Device

Source: https://midscenejs.com/integrate-with-any-interface

Instantiate the Agent by passing a custom device implementation and configuration options.

```typescript
const device = new SampleDevice({
  deviceName: 'Demo Panel',
  width: 1920,
  height: 1080,
});

const agent = new Agent(device, {
  actionContext: 'Dismiss pop-ups before every task.',
  generateReport: true,
  customActions: [], // optional extra DeviceAction list
});
```

--------------------------------

### Define Automation Tasks

Source: https://midscenejs.com/yaml-script-runner

YAML configuration examples for driving web browsers, Android devices, and iOS devices.

```yaml
page:
  url: https://www.bing.com

tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - sleep: 3000
      - aiAssert: The results show weather information
```

```yaml
android:
  deviceId: s4ey59 # find the device id with `adb devices`

tasks:
  - name: Maps Navigation
    flow:
      - ai: Open the Maps app
      - ai: Input 'West Lake, Hangzhou' in the search bar, and click the search button
      - ai: Click the first search result, enter the details page
      - ai: Click "Directions" button, enter the route planning page
      - ai: Click "Start" button to start navigation
```

```yaml
ios:
  wdaPort: 8100

tasks:
  - name: Change System Settings
    flow:
      - ai: Open the Settings app
      - ai: Tap "Display & Brightness"
      - ai: Turn on "Dark Mode"
      - aiAssert: Dark Mode is enabled
```

--------------------------------

### Install Midscene dependencies

Source: https://midscenejs.com/bridge-mode

Install the required @midscene/web and tsx packages using your preferred package manager.

```bash
npm install @midscene/web tsx --save-dev
```

```bash
yarn add @midscene/web tsx --save-dev
```

```bash
pnpm add @midscene/web tsx --save-dev
```

```bash
bun add @midscene/web tsx --save-dev
```

```bash
deno add npm:@midscene/web npm:tsx --save-dev
```

--------------------------------

### Example test case with Midscene

Source: https://midscenejs.com/integrate-with-playwright

A complete example demonstrating AI-driven actions and assertions in a Playwright test.

```typescript
import { expect } from '@playwright/test';
import { test } from './fixture';

test.beforeEach(async ({ page }) => {
  page.setViewportSize({ width: 400, height: 905 });
  await page.goto('https://www.ebay.com');
  await page.waitForLoadState('networkidle');
});

test('search headphone on ebay', async ({
  ai,
  aiQuery,
  aiAssert,
  aiInput,
  aiTap,
  aiScroll,
  aiWaitFor,
  aiRightClick,
  recordToReport,
}) => {
  // Use aiInput to enter search keyword
  await aiInput('Headphones', 'search box');

  // Use aiTap to click search button
  await aiTap('search button');

  // Wait for search results to load
  await aiWaitFor('search results list loaded', { timeoutMs: 5000 });

  // Use aiScroll to scroll to bottom
  await aiScroll(
    {
      scrollType: 'untilBottom',
    },
    'search results list',
  );

  // Use aiQuery to get product information
  const items = await aiQuery<Array<{ title: string; price: number }>>(
    'get product titles and prices from search results',
  );

  console.log('headphones in stock', items);
  expect(items?.length).toBeGreaterThan(0);

  // Use aiAssert to verify filter functionality
  await aiAssert('category filter exists on the left side');

  // Use recordToReport to capture the current state
  await recordToReport('Search Results', {
    content: 'Final search results for headphones',
  });
});
```

--------------------------------

### Initialize Local Computer Agent

Source: https://midscenejs.com/reference

Connects to a local desktop display, showing both primary and specific display selection.

```typescript
import { agentForComputer } from '@midscene/computer';

// Connect to primary display
const agent = await agentForComputer({
  aiActContext: 'You are automating a desktop application.',
});

// Connect to specific display
const displays = await ComputerDevice.listDisplays();
const agent2 = await agentForComputer({
  displayId: displays[1].id,
});
```

--------------------------------

### agentForComputer(opts?)

Source: https://midscenejs.com/reference

Creates an agent for local desktop automation.

```APIDOC
## agentForComputer(opts?)

### Description
Creates an agent for local desktop automation. An alias `agentFromComputer` is also available.

### Parameters
#### Options (LocalComputerAgentOpt)
- **displayId** (string) - Optional - Display to control. Use ComputerDevice.listDisplays() to list available displays.
- **keyboardDriver** ('applescript' | 'libnut') - Optional - On macOS, keyboard event backend. Default: 'applescript'.
- **headless** (boolean) - Optional - On Linux, set to true to start a virtual display with Xvfb. Default: false.
- **xvfbResolution** (string) - Optional - Resolution of the Xvfb virtual display. Default: '1920x1080x24'.
- **keyboardTypeDelay** (number) - Optional - Minimum delay in milliseconds between keystrokes.
- **aiActContext** (string) - Optional - Context for AI actions.

### Request Example
import { agentForComputer } from '@midscene/computer';

const agent = await agentForComputer({
  aiActContext: 'You are automating a desktop application.',
});
```

--------------------------------

### Write a desktop automation script

Source: https://midscenejs.com/platforms/desktop

Initialize the computer agent and perform basic actions like querying screen info, moving the mouse, and asserting content.

```typescript
import { agentForComputer } from '@midscene/computer';

(async () => {
  // Create an agent
  const agent = await agentForComputer({
    aiActionContext: 'You are controlling a desktop computer.',
  });

  // Take a screenshot and query information
  const screenInfo = await agent.aiQuery(
    '{width: number, height: number}, get screen resolution'
  );
  console.log('Screen resolution:', screenInfo);

  // Move mouse to center
  await agent.aiAct('move mouse to center of screen');

  // Assert screen has content
  await agent.aiAssert('The screen has visible content');

  console.log('Desktop automation completed!');
})();
```

--------------------------------

### YAML structure example

Source: https://midscenejs.com/automate-with-scripts-in-yaml.html

Basic structure for defining a task flow.

```yaml
- name: <name>
  flow:
    # ...
```

--------------------------------

### Configure Playwright download host

Source: https://midscenejs.com/integrate-with-playwright

Use a mirror to speed up the installation of Playwright browser binaries on limited networks.

```bash
PLAYWRIGHT_DOWNLOAD_HOST="https://npmmirror.com/mirrors/playwright" npx playwright install
```

--------------------------------

### Terminate App Example

Source: https://midscenejs.com/reference

Force-stop a specific Android application by package name.

```typescript
await agent.terminate('com.android.settings');
```
Tokens:
Raw