← 返回阅读笔记 / Back to reading note
来源:作者原文。2026-09-07抓取,原始行列,不是统一复跑成绩。
| Initial State | Task Instruction | Evaluation Script (Simplified) |
![]() |
Can you help me clean up my computer by getting rid of all the cookies that Amazon might have saved? | cookie_data = get_cookie_data(env) |
| rule = {"type":"domains", | ||
| "domains":[".amazon.com"]} | ||
| is_cookie_deleted(cookie_data, rule) | ||
![]() |
Rename “Sheet 1” to “LARS Resources”. Then make a copy of it. Place the copy before “Sheet 2” and rename it by appending a suffix “(Backup)”, … | result = get_file(env) |
| expected = get_file(cloud) | ||
| rules = [{"type":"sheet_name"}, | ||
| {"type":"sheet_data", | ||
| "sheet_idx0":0, | ||
| "sheet_idx1":1}...] | ||
| compare_table(result, expected, rules) | ||
![]() |
I’ve drafted an e-mail reminder for those who haven’t paid tuition. Please help me to check out their e-mails from the payment record and add to the receiver field. | tree = get_a11y_tree(env) |
| rules = [{"selectors": | ||
| ["tool-bar[attr|id=MsgHeadersToolbar] | ||
| label[name=To] | ||
| [attr|class=\"address-pill\"]> | ||
| label[attr|class=\"pill-label\"] | ||
| [name*=\"fox@someuniversity.edu...] | ||
| check_a11y_tree(tree, rules) |
| Function | Description |
| moveTo(x, y) | Moves the mouse to the specified coordinates. |
| click(x, y) | Clicks at the specified coordinates. |
| write(‘text’) | Types the specified text at the current cursor location. |
| press(‘enter’) | Presses the Enter key. |
| hotkey(‘ctrl’, ‘c’) | Performs the Ctrl+C hotkey combination (copy). |
| scroll(200) | Scrolls up by 200 units. |
| scroll(-200) | Scrolls down by 200 units. |
| dragTo(x, y) | Drags the mouse to the specified coordinates. |
| keyDown(‘shift’) | Holds down the Shift key. |
| keyUp(‘shift’) | Releases the Shift key. |
| WAIT | Agent decides it should wait. |
| FAIL | Agent decides the task is infeasible. |
| DONE | Agent decides the task is finished. |
| Statistic | Number |
| Total tasks (Ubuntu) | 369 (100%) |
| - Multi-App Workflow | 101 (27.4%) |
| - Single-App | 268 (72.6%) |
| - Integrated | 84 (22.8%) |
| - Infeasible | 30 (8.1%) |
| Supp. tasks (Windows) | 43 |
| Initial States | 302 |
| Eval. Scripts | 134 |
| Statistic | Number |
| Total tasks (Ubuntu) | 369 (100%) |
| - Multi-App Workflow | 101 (27.4%) |
| - Single-App | 268 (72.6%) |
| - Integrated | 84 (22.8%) |
| - Infeasible | 30 (8.1%) |
| Supp. tasks (Windows) | 43 |
| Initial States | 302 |
| Eval. Scripts | 134 |
| # Instances (# Templates) | Control. Exec. Env.? | Environment Scalability? | Multimodal Support? | Cross- App? | Intermediate Init. State? | # Exec.-based Eval. Func. | |
| GAIA [36] | 466 | ✗ | - | ✗ | ✗ | ✗ | 0 |
| Mind2Web [9] | 2350 | ✗ | - | ✓ | ✗ | ✓ | 0 |
| WebLINX [33] | 2337 | ✗ | - | ✓ | ✗ | ✓ | 0 |
| PixelHelp [27] | 187 | ✗ | - | ✓ | ✗ | ✗ | 0 |
| MetaGUI [47] | 1125 | ✗ | - | ✓ | ✗ | ✗ | 0 |
| AitW [40] | 30 | ✗ | - | ✓ | ✗ | ✓ | 0 |
| OmniAct [21] | 9802 | ✗ | - | ✓ | ✗ | ✓ | 0 |
| AgentBench [32] | 1091 | Multi-isolated | ✗ | ✗ | ✗ | ✗ | 7 |
| InterCode [57] | 1350 (3) | Code | ✗ | ✗ | ✗ | ✗ | 3 |
| MiniWoB++ [30] | 125 | Web | ✗ | ✓ | ✗ | ✗ | 125 |
| WebShop [58] | 12 (1) | Web | ✗ | ✓ | ✗ | ✗ | 1 |
| WebArena [66] | 812 (241) | Web | ✗ | ✓ | ✗ | ✗ | 5 |
| VWebArena [22] | 910 (314) | Web | ✗ | ✓ | ✗ | ✗ | 6 |
| WorkArena [10] | 23 (29) | Web | ✗ | ✓ | ✗ | ✓ | 7 |
| WikiHow [61] | 150 (16) | Mobile | ✗ | ✓ | ✗ | ✗ | 16 |
| AssistGUI [13] | 100 | ✗ | ✗ | ✓ | ✗ | ✓ | 2 |
| OSWorld | 369 | Computer | ✓ | ✓ | ✓ | ✓ | 134 |
| Inputs | Model | Success Rate (↑) | |||||
| OS | Office | Daily | Profess. | Workflow | Overall | ||
| A11y tree | Mixtral-8x7B | 12.50% | 1.01% | 4.79% | 6.12% | 0.09% | 2.98% |
| Llama-3-70B | 4.17% | 1.87% | 2.71% | 0.00% | 0.93% | 1.61% | |
| GPT-3.5 | 4.17% | 4.43% | 2.71% | 0.00% | 1.62% | 2.69% | |
| GPT-4 | 20.83% | 3.58% | 25.64% | 26.53% | 2.97% | 12.24% | |
| Gemini-Pro | 4.17% | 1.71% | 3.99% | 4.08% | 0.63% | 2.37% | |
| Gemini-Pro-1.5 | 12.50% | 2.56% | 7.83% | 4.08% | 3.60% | 4.81% | |
| Qwen-Max | 29.17% | 3.58% | 8.36% | 10.20% | 2.61% | 6.87% | |
| GPT-4o | 20.83% | 6.99% | 16.81% | 16.33% | 7.56% | 11.36% | |
| Screenshot | CogAgent | 4.17% | 0.85% | 2.71% | 0.00% | 0.00% | 1.11% |
| GPT-4V | 12.50% | 1.86% | 7.58% | 4.08% | 6.04% | 5.26% | |
| Gemini-ProV | 8.33% | 3.58% | 6.55% | 16.33% | 2.08% | 5.80% | |
| Gemini-Pro-1.5 | 12.50% | 6.99% | 2.71% | 6.12% | 3.60% | 5.40% | |
| Claude-3-Opus | 4.17% | 1.87% | 2.71% | 2.04% | 2.61% | 2.42% | |
| GPT-4o | 8.33% | 3.58% | 6.07% | 4.08% | 5.58% | 5.03% | |
| Screenshot | CogAgent | 4.17% | 0.85% | 2.71% | 0.62% | 0.09% | 1.32% |
| + A11y tree | GPT-4V | 16.66% | 6.99% | 24.50% | 18.37% | 4.64% | 12.17% |
| Gemini-ProV | 4.17% | 4.43% | 6.55% | 0.00% | 1.52% | 3.48% | |
| Gemini-Pro-1.5 | 12.50% | 3.58% | 7.83% | 8.16% | 1.52% | 5.10% | |
| Claude-3-Opus | 12.50% | 3.57% | 5.27% | 8.16% | 1.00% | 4.41% | |
| GPT-4o | 41.67% | 6.16% | 12.33% | 14.29% | 7.46% | 11.21% | |
| Set-of-Mark | CogAgent | 4.17% | 0.00% | 2.71% | 0.00% | 0.53% | 0.99% |
| GPT-4V | 8.33% | 8.55% | 22.84% | 14.28% | 6.57% | 11.77% | |
| Gemini-ProV | 4.17% | 1.01% | 1.42% | 0.00% | 0.63% | 1.06% | |
| Gemini-Pro-1.5 | 16.67% | 5.13% | 12.96% | 10.20% | 3.60% | 7.79% | |
| Claude-3-Opus | 12.50% | 2.72% | 14.24% | 6.12% | 4.49% | 6.72% | |
| GPT-4o | 20.83% | 3.58% | 3.99% | 2.04% | 3.60% | 4.59% | |
| Human Performance | 75.00% | 71.79% | 70.51% | 73.47% | 73.27% | 72.36% | |
| Task Subset | % of Total | SR (↑) |
| Easy | 28.72% | 16.78% |
| Medium | 40.11% | 13.12% |
| Hard | 30.17% | 4.59% |
| Infeasible | 8.13% | 16.67% |
| Feasible | 91.87% | 13.34% |
| Single-App | 72.63% | 13.74% |
| Multi-App Workflow | 27.37% | 6.57% |
| OS | SR (%) | Correlation Coefficient |
| Ubuntu | 4.88 | 0.7 |
| Windows | 2.55 |
| Action Type | Parameters | Note |
| MOVE_TO | x, y | Move the cursor to the specified position |
| CLICK |
button,
x, y, num_clicks |
Click the left button if the button not specified, otherwise click the specified button; click at the current position if x and y are not specified, otherwise click at the specified position |
| MOUSE_DOWN | button | Press the left button if the button not specified, otherwise press the specified button |
| MOUSE_UP | button | Release the left button if the button not specified, otherwise release the specified button |
| RIGHT_CLICK | x, y | Right click at the current position if x and y are not specified, otherwise right click at the specified position |
| DOUBLE_CLICK | x, y | Double click at the current position if x and y are not specified, otherwise double click at the specified position |
| DRAG_TO | x, y | Drag the cursor to the specified position with the left button pressed |
| SCROLL | dx, dy | Scroll the mouse wheel up or down |
| TYPING | text | Type the specified text |
| PRESS | key | Press the specified key and release it |
| KEY_DOWN | key | Press the specified key |
| KEY_UP | key | Release the specified key |
| HOTKEY | keys | Press the specified key combination |
| WAIT | - | Wait until the next action |
| FAIL | - | Decide the task cannot be performed |
| DONE | - | Decide the task is done |
| OS | Calc | Impress | Writer | VLC | Thunderbird | Chrome | VSCode | GIMP | Workflow | Overall | |
| Examples | 24 | 47 | 47 | 23 | 17 | 15 | 46 | 23 | 26 | 101 | 369 |
| Avg. Inst. Tokens | 22.38 | 33.30 | 25.19 | 35.30 | 35.82 | 34.07 | 22.07 | 20.78 | 16.23 | 51.24 | 33.36 |
| #Infeasible | 5 | 1 | 0 | 1 | 3 | 1 | 3 | 5 | 10 | 1 | 30 |
| #Integrated | 7 | 19 | 30 | 0 | 0 | 0 | 26 | 0 | 0 | 2 | 84 |
| Excel | Word | PPT | Workflow | Overall | |
| Examples | 11 | 9 | 7 | 16 | 43 |
| Avg. Inst. Tokens | 19.45 | 21.44 | 21.86 | 47.57 | 32.48 |
| Related App(s) | Instruction(s) | Screenshot | Abilities Needed |
|---|---|---|---|
| OS | I want to install Spotify on my current system. Could you please help me? |
|
knowledge of OS; omit distractions |
| Calc | Check the names in column “Names with duplicates” and put the unique ones in column “Unique Names”. Keep the original order. |
|
massive elements; knowledge tricks or reasoning over long actions |
| Calc | I have a lookup table for the officers of each branch. Please, here is another table in which I need to fill with the officer names according the headoffice (i.e., the branch name). Help me to complete this. |
|
massive elements; knowledge of formulas and functions |
| Calc | Given a partial calendar, please highlight all the weekends (Saturday & Sunday) by setting the cell background as red (#ff0000). |
|
massive elements; commonsense reasoning; software tricks |
| Impress | I closed the slide pannel on the left and idk how to get it back please help |
|
software knowledge; imagine about UI layouts; overcome typos in instruction |
| Impress | On it Whenever I launch a LibreOffice Impress, it uses both screens, one for current slide and next slide and another for actual presentation. What I want is to use only one monitor which shows presentation. I dont want the screen with Current slide and Next slide so that it can be used for other purposes. How should I achieve this? |
|
reason from unprofessional phenomenon expression |
| Writer | Copy the screenshot 1.png from the desktop to where my cursor is located |
|
locate the position of cursor; switch from desktop and app |
| Chrome | Can you help me clean up my computer by getting rid of all the tracking things that Amazon might have saved? I want to make sure my browsing is private and those sites don’t remember me. |
|
understanding the unprofessional expression |
| VLC | I am reading lecture note in PDF while a music video is running in VLC media player. But I find I need to switch to the player every time I need to pause/start.Could you help me change the setting to allow pausing the video using keyboard shortcut without minimizing the PDF reader? I want to focus on the lecture note and don’t be disturbed by the app switching. |
|
understanding the reference from unprofessional expression; software knowledge |
| VLC | Hey, could you turn this video the right way up for me? And once it’s flipped around, could you save it for me with the name ‘1984_Apple.mp4’ on the main screen where all my files are? |
|
software knowledge; spatial judgment ability |
| Thunderbird | Create a local folder called "Promotions" and create a filter to auto move the inbox emails whose subject contains “discount” to the new folder |
|
software knowledge |
| Thunderbird | Attach the my AWS bill to the email. The bill is stored at /aws-bill.pdf. Don’t close it or send it. I haven’t finish all the contents. |
|
file management; extra requirement |
| Thunderbird | I’ve got a bunch of email accounts in Thunderbird, and it’s a hassle to check them one by one. Can you show me how to set up a unified inbox so I can see all my emails in one place? |
|
deep-hided feature, need to be explored even by human users; pop-up window |
| VS Code | Please modify VS Code’s settings to disable error reporting for Python missing imports. |
|
software knowledge to deal with settings; reasoning to understand the cause and solution of the error |
| VS Code | Please help me install the autoDocstring extension in VS Code. |
|
software knowledge to deal with Extensions; reasoning to search and install the extension successfully |
| GIMP | Could you make the background of this image transparent for me? |
|
precise and intricate operations |
| GIMP | Help me choose the yellow triangle and position it at the center of my picture. |
|
spatial perception and reasoning, as well as precise control of actions |
| Multiple (VLC+GIMP) | Could you help me create an Animated GIF from a video file using VLC and GIMP from the source of video “src.mp4”, 5-second clip beginning at 00:03? |
|
software knowledge to undergo sophisticated processes and ability to process multi-step procedure successfully |
| Multiple (ThunderBird+ Writer+Chrome) | Help me export charts, graph or other images from docx files received in email “Lecture Document” in Notes folder and upload these png files to the figures/ folder in Google Drive for later use (use numbers to name them). |
|
ability to selectively export charts, graphs and images from docx file; software knowledge for google drive file upload |
| Multiple (Chrome+Calc) | Could you help me extract data in the table from a new invoice uploaded to my Google Drive, then export it to a Libreoffice calc .xlsx file in the desktop? |
|
ability to do table data extraction; export data to .xlsx file |
| Condition | Ubuntu | Windows |
|---|---|---|
| Node Tags | document, item, button, heading, label, scrollbar, searchbox, textbox, link, tabelement, textfield, textarea, menu, alert, canvas, check-box, combo-box, entry, icon, image, paragraph, scroll-bar, section, slider, static, table-cell, terminal, text, netuiribbontab, start, trayclockwclass, traydummysearchcontrol, uiimage, uiproperty, uiribboncommandbar | Same as Ubuntu |
| Showing | True | Not Applicable |
| Visible | True | True |
| Enabled or Editable or Expandable or Checkable | True | True |
| Has Name or Text or Image | True | True |
| Position | ||
| Size |
| Inputs | Model | Success Rate (↑) | |||||||||
| OS | Calc | Impress | Writer | VLC | TB | Chrome | VSC | GIMP | Workflow | ||
| A11y | Mixtral-8x7B | 12.50 | 0.00 | 0.39 | 4.34 | 10.22 | 6.67 | 2.17 | 8.69 | 3.85 | 0.10 |
| GPT-3.5 | 4.17 | 2.13 | 6.77 | 4.35 | 6.53 | 0.00 | 2.17 | 0.00 | 0.00 | 1.62 | |
| Gemini-Pro | 4.17 | 0.00 | 2.13 | 4.35 | 12.41 | 0.00 | 2.17 | 0.00 | 7.69 | 0.63 | |
| GPT-4 | 20.83 | 0.00 | 6.77 | 4.35 | 23.53 | 26.67 | 26.09 | 30.43 | 23.08 | 2.97 | |
| Gemini-Pro-1.5 | 12.50 | 2.13 | 2.13 | 4.35 | 6.53 | 0.00 | 10.87 | 8.70 | 0.00 | 3.60 | |
| Llama-3-70B | 4.17 | 0.00 | 0.39 | 8.70 | 6.53 | 0.00 | 2.17 | 0.00 | 0.00 | 0.63 | |
| GPT-4o | 20.83 | 6.38 | 6.77 | 8.69 | 12.41 | 20.00 | 17.39 | 21.74 | 11.54 | 7.56 | |
| Qwen-Max | 29.17 | 0.00 | 2.52 | 13.04 | 8.95 | 0.00 | 10.87 | 8.70 | 11.54 | 2.61 | |
| Screen | CogAgent | 4.17 | 0.00 | 0.00 | 4.34 | 6.53 | 0.00 | 2.17 | 0.00 | 0.00 | 0.00 |
| Gemini-ProV | 8.33 | 0.00 | 6.77 | 4.35 | 12.41 | 0.00 | 6.52 | 8.70 | 23.08 | 2.08 | |
| GPT-4V | 12.50 | 0.00 | 2.52 | 4.35 | 18.34 | 0.00 | 6.52 | 0.00 | 7.69 | 6.04 | |
| Claude-3-Opus | 4.17 | 0.00 | 2.52 | 4.34 | 6.53 | 0.00 | 2.17 | 0.00 | 3.84 | 2.61 | |
| Gemini-Pro-1.5 | 12.50 | 0.00 | 13.16 | 8.70 | 6.53 | 0.00 | 2.17 | 0.00 | 11.54 | 3.60 | |
| GPT-4o | 8.33 | 0.00 | 6.77 | 4.35 | 16.10 | 0.00 | 4.35 | 4.35 | 3.85 | 5.58 | |
| Screen | CogAgent | 4.17 | 2.17 | 0.00 | 4.35 | 6.53 | 0.00 | 2.17 | 0.00 | 0.00 | 0.10 |
| + A11y | Gemini-ProV | 4.17 | 2.13 | 6.77 | 4.35 | 18.30 | 0.00 | 4.35 | 0.00 | 0.00 | 1.52 |
| GPT-4V | 16.67 | 0.00 | 6.77 | 21.73 | 24.18 | 33.33 | 21.74 | 21.74 | 15.38 | 4.59 | |
| Claude-3-Opus | 12.50 | 2.13 | 4.65 | 4.34 | 18.30 | 0.00 | 2.17 | 8.69 | 7.69 | 0.99 | |
| Gemini-Pro-1.5 | 12.50 | 0.00 | 4.65 | 8.70 | 12.41 | 0.00 | 8.70 | 4.35 | 11.54 | 1.56 | |
| GPT-4o | 41.67 | 4.26 | 6.81 | 8.70 | 9.50 | 6.67 | 15.22 | 30.43 | 0.00 | 7.46 | |
| SoM | CogAgent | 4.17 | 2.17 | 0.00 | 4.34 | 6.53 | 0.00 | 2.17 | 0.00 | 0.00 | 0.00 |
| Gemini-ProV | 4.17 | 0.00 | 0.39 | 4.34 | 6.53 | 0.00 | 0.00 | 0.00 | 0.00 | 0.63 | |
| GPT-4V | 8.33 | 8.51 | 6.38 | 13.04 | 10.66 | 33.33 | 23.91 | 17.39 | 11.54 | 6.57 | |
| Claude-3-Opus | 12.50 | 2.13 | 0.39 | 8.70 | 6.53 | 13.33 | 17.39 | 0.00 | 11.54 | 4.49 | |
| Gemini-Pro-1.5 | 16.67 | 0.00 | 10.64 | 4.35 | 24.18 | 13.33 | 8.70 | 4.35 | 15.38 | 3.60 | |
| GPT-4o | 20.83 | 0.00 | 6.77 | 4.35 | 6.53 | 0.00 | 4.35 | 4.35 | 0.00 | 3.60 | |
| Human Performance | 75.00 | 61.70 | 80.85 | 73.91 | 70.59 | 46.67 | 78.26 | 73.91 | 73.08 | 73.27 | |