### Execute Commands with MobileWorld and Qwen3.5 Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/blog_posts/MobileWorld Update Can Frontier Models Really Control Your Phone.md Execute a command using MobileWorld with the Qwen3.5 model hosted on Dashscope. This example demonstrates setting an alarm on the device. ```bash uv run mw test "set an alarm at 8:00 am" \ --agent-type general_e2e \ --model_name qwen3.5-plus \ --llm_base_url https://dashscope.aliyuncs.com/compatible-mode/v1 \ --aw-host http://127.0.0.1:6800 \ --api_key xxx ``` -------------------------------- ### Get Arena Screenshot Blocks Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/arena.html Selects all screenshot blocks within the arena comparison columns. ```javascript function arenaScreenshotBlocks() { return document.querySelectorAll( '#arena-col-a-body .traj-screenshot-block, #arena-col-b-body .traj-screenshot-block' ); } ``` -------------------------------- ### Bundle Trajectories with Screenshots Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/index.html This command bundles trajectory logs, including screenshots, into a compressed JSON file. Replace `` with your specific run name. ```bash python site/bundle_trajs.py traj_logs// --with-screenshots ``` -------------------------------- ### Initialize Syntax Highlighting Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/index.html Initializes the highlight.js library to apply syntax highlighting to code blocks on the page. This should be called after the DOM is fully loaded. ```javascript hljs.highlightAll(); ``` -------------------------------- ### Render Task Steps Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/index.html Iterates through task steps to generate HTML for display. Includes logic for action types, coordinates, text, direction, frame index, user responses, and tool calls. ```javascript const html = steps.map((step, i) => { const pred = step.pred || ''; const think = step.think || ''; const response = step.response || ''; const meta = task.meta; const videoPromise = task.video_promise; const trajSteps = document.getElementById('traj-steps'); const trajTaskSelect = document.getElementById('traj-task-select'); const trajModal = document.getElementById('traj-modal'); const trajModalClose = document.getElementById('traj-modal-close'); const trajModalBg = document.getElementById('traj-modal-bg'); const taskNames = Object.keys(tasks); let renderGeneration = 0; function renderTask(taskName) { const task = tasks[taskName]; const steps = task.steps; const meta = task.meta; const videoPromise = task.video_promise; let html = ''; const action = step.action || {}; let actionStr = action.action_type || 'unknown'; if (action.x !== undefined && action.y !== undefined) { actionStr += ` (${action.x}, ${action.y})`; } if (action.text) { actionStr += `: "${action.text}"`; } if (action.direction) { actionStr += ` ${action.direction}`; } const hasFrame = step.frame_index !== undefined && meta && meta.video_file; const screenshotHtml = hasFrame ? `
Screenshot
Loading screenshot…
` : ''; const askUser = step.ask_user_response ? `
User Response${escHtml(step.ask_user_response)}
` : ''; const toolCall = step.tool_call ? `
Tool Call
${escHtml(JSON.stringify(step.tool_call, null, 2))}
` : ''; return `
Step ${step.step || i + 1} ${escHtml(actionStr)}
${screenshotHtml} ${think ? `
Thinking
${escHtml(think)}
` : ''} ${response ? `
${escHtml(response)}
` : ''} ${askUser} ${toolCall}
`; }).join(''); if (task.token_usage) { const tu = task.token_usage; html += `
Token usage: ${(tu.prompt_tokens || 0).toLocaleString()} prompt · ${(tu.completion_tokens || 0).toLocaleString()} completion · ${(tu.total_tokens || 0).toLocaleString()} total
`; } trajSteps.innerHTML = html; updateExpandAllLabel(); if (videoPromise && meta) { const myGen = ++renderGeneration; (async () => { const videoEl = await videoPromise; if (myGen !== renderGeneration) return; if (!videoEl) { trajSteps.querySelectorAll('.traj-screenshot.is-loading .traj-screenshot-spinner').forEach(el => { el.innerHTML = ' Screenshot unavailable'; }); return; } const canvases = trajSteps.querySelectorAll('.traj-canvas'); for (const canvas of canvases) { if (myGen !== renderGeneration) return; const fi = parseInt(canvas.dataset.frame); const si = parseInt(canvas.dataset.step); const stepAction = steps[si]?.action || null; await renderScreenshotCanvas(canvas, videoEl, fi, meta.fps, meta, stepAction); canvas.parentElement?.classList.remove('is-loading'); } })(); } } trajTaskSelect.onchange = () => renderTask(trajTaskSelect.value); if (taskNames.length > 0) renderTask(taskNames[0]); ``` -------------------------------- ### Model-Specific Prompt Adaptation for Seed-2.0-Pro Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/blog_posts/MobileWorld Update Can Frontier Models Really Control Your Phone.md This snippet shows an adaptation of a prompt from the OSWorld repository for the Seed-2.0-Pro model. It is designed to better match the expected input format for this specific model. ```python from mm_agents.seed_agent import SeedAgent # ... other imports and setup ... agent = SeedAgent(model_name="seed-2.0-pro", ...) ``` -------------------------------- ### Trajectory Viewer Initialization Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/index.html Initializes elements for the trajectory viewer modal and sets up event listeners for closing it. ```javascript const trajModal = document.getElementById('traj-modal'); const trajModalClose = document.getElementById('traj-modal-close'); const trajModalBg = trajModal.querySelector('.modal-background'); const trajModelName = document.getElementById('traj-model-name'); const trajTaskSelect = document.getElementById('traj-task-select'); const trajResultBadge = document.getElementById('traj-result-badge'); const trajLoading = document.getElementById('traj-loading'); const trajSteps = document.getElementById('traj-steps'); const trajCache = {}; ``` -------------------------------- ### Render Step Details Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/arena.html Generates HTML for a single step in the Arena trajectory, including action details, screenshots, predictions, user responses, and tool calls. Use this to display individual steps within a comparison. ```javascript function renderStep(i, step, side, pred, meta) { const action = step.action; let actionStr = ''; if (action.x !== undefined && action.y !== undefined) actionStr += ` (${action.x}, ${action.y}) `; if (action.text) actionStr += `: "${action.text}" `; if (action.direction) actionStr += ` ${action.direction} `; const hasFrame = step.frame_index !== undefined && meta && meta.video_file; const screenshotHtml = hasFrame ? `
Screenshot
Loading screenshot…
` : ''; const askUser = step.ask_user_response ? `
User Response${escHtml(step.ask_user_response)}
` : ''; const toolCall = step.tool_call ? `
Tool Call
${escHtml(JSON.stringify(step.tool_call, null, 2))}
` : ''; return `
Step ${step.step || i + 1} ${escHtml(actionStr)}
${screenshotHtml} ${pred ? `
${escHtml(pred)}
` : ''} ${askUser} ${toolCall}
`; }).join('') } ``` -------------------------------- ### Run Evaluation Command Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/index.html Use this command to run the evaluation for your agent. Ensure you replace `` with the actual module name of your agent. ```bash uv run mw eval --agent --tasks all ``` -------------------------------- ### Apply URL Parameters for Step Navigation Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/arena.html Parses URL parameters to find and highlight a specific task step. Scrolls the step into view with a smooth animation and a temporary highlight. Handles invalid or out-of-range step numbers. ```javascript const applyUrlParams = async () => { const params = new URLSearchParams(window.location.search); const task = params.get('task'); const stepRaw = params.get('step'); if (task == null) { // No task specified, nothing to do. return; } // If step is specified, scroll it into view + flash highlight. if (stepRaw != null) { const stepIdx = parseInt(stepRaw, 10) - 1; // URL is 1-indexed if (!Number.isFinite(stepIdx) || stepIdx < 0) { console.warn(`[arena] Invalid step in URL: ${stepRaw}`); return; } // Prefer the left (Model A) pane; fall back to right. let stepEl = document.querySelector( `#arena-col-a-body .traj-step[data-step="${stepIdx}"]` ); if (!stepEl) { stepEl = document.querySelector( `#arena-col-b-body .traj-step[data-step="${stepIdx}"]` ); } if (!stepEl) { console.warn(`[arena] Step ${stepRaw} out of range for task ${task}`); return; } // Defer slightly so layout settles after renderComparison. setTimeout(() => { stepEl.scrollIntoView({ block: 'center', behavior: 'smooth' }); stepEl.classList.add('url-target-step'); setTimeout(() => stepEl.classList.remove('url-target-step'), 2000); }, 100); } } } applyUrlParams().catch(err => { console.warn('[arena] applyUrlParams failed:', err); }); ``` -------------------------------- ### General End-to-End Prompt Structure Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/blog_posts/MobileWorld Update Can Frontier Models Really Control Your Phone.md This prompt structure is designed for direct coordinate output and compatibility across various LLMs. It uses a structured action space with JSON schemas and a Thought-Action format for parsing and debugging. ```python { "action_type": "click", "coordinate": [x, y] } ``` -------------------------------- ### Configure ADB PATH on macOS Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/blog_posts/MobileWorld Update Can Frontier Models Really Control Your Phone.md Add the ADB platform-tools directory to your system's PATH environment variable on macOS. This allows you to run ADB commands from any terminal location. ```bash # macOS — assuming the extracted directory is ~/Downloads/platform-tools export PATH=${PATH}:~/Downloads/platform-tools ``` -------------------------------- ### Render Comparison View Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/arena.html Renders the detailed comparison view for a given task, including goals, model names, step counts, and trajectory steps. It also initiates the rendering of side canvases and token usage. ```javascript function renderComparison(taskName) { const comp = document.getElementById('arena-comparison'); comp.style.display = ''; document.getElementById('arena-comparison-title').textContent = taskName; const taskA = dataA[taskName]; const taskB = dataB[taskName]; // Task goal const goalEl = document.getElementById('arena-task-goal'); const goalA = taskA?.traj?.[0]?.task_goal; const goalB = taskB?.traj?.[0]?.task_goal; const goal = goalA || goalB; goalEl.style.display = goal ? '' : 'none'; if (goal) goalEl.innerHTML = `Task Goal${escHtml(goal)}`; // Column headers document.getElementById('arena-col-a-name').textContent = modelInfoA.model; document.getElementById('arena-col-b-name').textContent = modelInfoB.model; const oA = taskOutcomes[taskName].a; const oB = taskOutcomes[taskName].b; setBadge('arena-col-a-badge', oA); setBadge('arena-col-b-badge', oB); document.getElementById('arena-col-a-steps').textContent = `${(taskA?.traj || []).length} steps`; document.getElementById('arena-col-b-steps').textContent = `${(taskB?.traj || []).length} steps`; // Render steps document.getElementById('arena-col-a-body').innerHTML = renderSteps(taskA, 'a'); document.getElementById('arena-col-b-body').innerHTML = renderSteps(taskB, 'b'); updateArenaExpandAllLabel(); // Render screenshot frames. Each side awaits its own video promise // so screenshots can appear independently as videos arrive. // The generation counter prevents stale frames from being painted // into a different task's canvases if the user switches tasks // before the video finishes downloading. const myGen = ++arenaRenderGeneration; async function renderSideCanvases(containerId, videoPromise, meta, task) { if (!videoPromise || !meta || !task) return; const container = document.getElementById(containerId); const vid = await videoPromise; if (myGen !== arenaRenderGeneration) return; if (!vid) { container.querySelectorAll('.traj-screenshot.is-loading .traj-screenshot-spinner').forEach(el => { el.innerHTML = ' Screenshot unavailable'; }); return; } const canvases = container.querySelectorAll('.traj-canvas'); for (const canvas of canvases) { if (myGen !== arenaRenderGeneration) return; const fi = parseInt(canvas.dataset.frame); const si = parseInt(canvas.dataset.step); const action = task.traj[si]?.action || null; await renderScreenshotCanvas(canvas, vid, fi, meta.fps, meta, action); canvas.parentElement?.classList.remove('is-loading'); } } renderSideCanvases('arena-col-a-body', videoPromiseA, trajMetaA, taskA); renderSideCanvases('arena-col-b-body', videoPromiseB, trajMetaB, taskB); // Token usage renderTokens('arena-col-a-tokens', taskA?.token_usage); renderTokens('arena-col-b-tokens', taskB?.token_usage); comp.scrollIntoView({ behavior: 'smooth', block: 'start' }); } ``` -------------------------------- ### Open Detail Modal Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/index.html Opens a modal window to display detailed information about a selected leaderboard row. It populates the modal header and body with model-specific data. ```javascript function openDetailModal(row) { const overallScore = getFilteredScore(row); const na = ''; const categoryLabel = row.category; modalHeader.innerHTML = ` ${categoryLabel}

${row.model}

${row.organization} · ${formatDate(row.date)}

`; function scoreRow(label, value) { const display = value !== null && value !== undefined ? value.toFixed(1) : null; const pct = display !== null ? Math.min(value / 100 * 100, 100) : 0; return `
${label} ${display !== null ? display : na}
`; } const linkDisplay = row.link ? (() => { try { return new URL(row.link).hostname; } catch { return row.link; } })() : null; modalBody.innerHTML = `
${overallScore !== null ? overallScore.toF` } // Note: This snippet appears truncated in the source. Only the provided code is included. ] // Note: This snippet appears truncated in the source. Only the provided code is included. } // Note: This snippet appears truncated in the source. Only the provided code is included. ] // Note: This snippet appears truncated in the source. Only the provided code is included. } // Note: This snippet appears truncated in the source. Only the provided code is included. ``` -------------------------------- ### Apply URL Parameters for Deep Linking Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/arena.html Parses URL parameters to configure the Arena interface, including model selection, task filtering, and step navigation. This function is crucial for sharing specific Arena states. ```javascript async function applyUrlParams() { const params = new URLSearchParams(window.location.search); const modelA = params.get('modelA') || params.get('model'); const modelB = params.get('modelB'); const task = params.get('task'); const filter = params.get('filter'); const category = params.get('category'); const stepRaw = params.get('step'); const knownModel = (n) => n && modelsWithTraj.some(m => m.model === n); if (modelA) { if (knownModel(modelA)) selA.value = modelA; else console.warn(`[arena] Unknown modelA in URL: ${modelA}`); } if (modelB) { if (knownModel(modelB)) selB.value = modelB; else console.warn(`[arena] Unknown modelB in URL: ${modelB}`); } if (!selA.value || !selB.value || selA.value === selB.value) return; try { await onModelChange(); } catch (err) { console.warn(' [arena] onModelChange failed during URL-param expansion:', err); return; } if (document.getElementById('arena-content').style.display === 'none') return; if (filter) { if (['pp', 'pf', 'fp', 'ff'].includes(filter)) { currentFilter = filter; highlightMatrixCell(currentFilter); } else { console.warn(`[arena] Invalid filter in URL: ${filter}`); } } if (category) { const catSelect = document.getElementById('arena-category-filter'); const hasCat = Array.from(catSelect.options).some(o => o.value === category); if (hasCat) catSelect.value = category; else console.warn(`[arena] Unknown category in URL: ${category}`); } renderTaskList(); if (task) { const safeTask = task.replace(/ ``` -------------------------------- ### Toggle All Trajectory Steps Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/index.html Manages the expand/collapse state of all trajectory screenshot blocks and updates the 'Expand all' button label accordingly. ```javascript const trajExpandAllBtn = document.getElementById('traj-expand-all'); function updateExpandAllLabel() { const blocks = trajSteps.querySelectorAll('.traj-screenshot-block'); if (blocks.length === 0) { trajExpandAllBtn.style.display = 'none'; return; } trajExpandAllBtn.style.display = ''; const allOpen = [...blocks].every(b => b.open); trajExpandAllBtn.textContent = allOpen ? 'Collapse all' : 'Expand all'; } trajExpandAllBtn.addEventListener('click', () => { const blocks = trajSteps.querySelectorAll('.traj-screenshot-block'); if (blocks.length === 0) return; const allOpen = [...blocks].every(b => b.open); blocks.forEach(b => { b.open = !allOpen; }); updateExpandAllLabel(); }); trajSteps.addEventListener('toggle', (e) => { if (e.target.classList?.contains('traj-screenshot-block')) updateExpandAllLabel(); }, true); ``` -------------------------------- ### Render Trajectory Steps Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/arena.html Generates HTML for rendering the steps of a trajectory for a given side (model). Handles cases with no trajectory data. ```javascript function renderSteps(task, side) { if (!task || !task.traj || task.traj.length === 0) { return '
No trajectory data.
'; } const meta = side === 'a' ? trajMetaA : trajMetaB; return task.traj.map((step, i) => { const pred = step.prediction || ''; const action = step.action || {}; let actionStr = action.action_type || 'unknown'; if (action.x ``` -------------------------------- ### Clone MobileWorld Repository Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/blog_posts/MobileWorld Update Can Frontier Models Really Control Your Phone.md Clone the latest version of the MobileWorld code from its GitHub repository. ```bash git clone https://github.com/Tongyi-MAI/MobileWorld.git ``` -------------------------------- ### Check Connected Android Devices with ADB Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/blog_posts/MobileWorld Update Can Frontier Models Really Control Your Phone.md Verify that your Android phone is successfully connected to your computer via USB and recognized by ADB. This command lists all attached devices. ```bash # Check connected devices ``` ```bash adb devices ``` -------------------------------- ### Loading and Caching Video Elements Source: https://github.com/tongyi-mai/mobileworld/blob/main/site/index.html Loads video elements from a URL, caches them for reuse, and ensures they are ready for playback. It handles metadata loading and potential errors. ```javascript // Point the