← 返回阅读笔记 / Back to reading note

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

SUP-A13 原表

来源:作者原文。2026-09-07抓取,原始行列,不是统一复跑成绩。

Table 1: Examples of our annotated evaluation scripts, which involve retrieving data from configuration files, the environment, and the cloud, and executing functions to assess functional correctness and obtain results. The example-wise evaluation facilitates the diversity of tasks and reliable evaluation of complex, real-world, open-ended tasks.
Initial State Task Instruction Evaluation Script (Simplified)
[Uncaptioned image] Can you help me clean up my computer by getting rid of all the cookies that Amazon might have saved? cookie_data = get_cookie_data(env)
rule = {"type":"domains",
"domains":[".amazon.com"]}
is_cookie_deleted(cookie_data, rule)
[Uncaptioned image] Rename “Sheet 1” to “LARS Resources”. Then make a copy of it. Place the copy before “Sheet 2” and rename it by appending a suffix “(Backup)”, … result = get_file(env)
expected = get_file(cloud)
rules = [{"type":"sheet_name"},
   {"type":"sheet_data",
   "sheet_idx0":0,
   "sheet_idx1":1}...]
compare_table(result, expected, rules)
[Uncaptioned image] I’ve drafted an e-mail reminder for those who haven’t paid tuition. Please help me to check out their e-mails from the payment record and add to the receiver field. tree = get_a11y_tree(env)
rules = [{"selectors":
   ["tool-bar[attr|id=MsgHeadersToolbar]
   label[name=To] 
   [attr|class=\"address-pill\"]>
   label[attr|class=\"pill-label\"]
   [name*=\"fox@someuniversity.edu...]
check_a11y_tree(tree, rules)
Table 2: Some examples of the mouse and keyboard actions 𝒜 \mathcal{A} in OSWorld . See App. A.3 for the complete list.
Function Description
moveTo(x, y) Moves the mouse to the specified coordinates.
click(x, y) Clicks at the specified coordinates.
write(‘text’) Types the specified text at the current cursor location.
press(‘enter’) Presses the Enter key.
hotkey(‘ctrl’, ‘c’) Performs the Ctrl+C hotkey combination (copy).
scroll(200) Scrolls up by 200 units.
scroll(-200) Scrolls down by 200 units.
dragTo(x, y) Drags the mouse to the specified coordinates.
keyDown(‘shift’) Holds down the Shift key.
keyUp(‘shift’) Releases the Shift key.
WAIT Agent decides it should wait.
FAIL Agent decides the task is infeasible.
DONE Agent decides the task is finished.
Table 3: Key statistics in OSWorld . The “Supp. tasks” refers to the Windows-based tasks, that could only be used after activation due to copyright restrictions.
Statistic Number
Total tasks (Ubuntu) 369 (100%)
- Multi-App Workflow 101 (27.4%)
- Single-App 268 (72.6%)
- Integrated 84 (22.8%)
- Infeasible 30 (8.1%)
Supp. tasks (Windows) 43
Initial States 302
Eval. Scripts 134
Table 3: Key statistics in OSWorld . The “Supp. tasks” refers to the Windows-based tasks, that could only be used after activation due to copyright restrictions.
Statistic Number
Total tasks (Ubuntu) 369 (100%)
- Multi-App Workflow 101 (27.4%)
- Single-App 268 (72.6%)
- Integrated 84 (22.8%)
- Infeasible 30 (8.1%)
Supp. tasks (Windows) 43
Initial States 302
Eval. Scripts 134
Table 4: Comparison of different environments for benchmarking digital agents. The columns indicate: the number of task instances and templates (if applicable) where the task instantiated from templates through configurations (# Instances (# Templates)), whether they provide a controllable executable environment (Control. Exec. Env.), the ease of adding new tasks involving arbitrary applications in open domains (Environment Scalability), support for multimodal agent evaluation (Multimodal Support), support for and inclusion of cross-app tasks (Cross-App), capability to start tasks from an intermediate initial state (Intermediate Init. State), and the number of execution-based evaluation functions (# Exec.-based Eval. Func.).
# Instances (# Templates) Control. Exec. Env.? Environment Scalability? Multimodal Support? Cross- App? Intermediate Init. State? # Exec.-based Eval. Func.
GAIA [36] 466 ✗ - ✗ ✗ ✗ 0
Mind2Web [9] 2350 ✗ - ✓ ✗ ✓ 0
WebLINX [33] 2337 ✗ - ✓ ✗ ✓ 0
PixelHelp [27] 187 ✗ - ✓ ✗ ✗ 0
MetaGUI [47] 1125 ✗ - ✓ ✗ ✗ 0
AitW [40] 30kk ✗ - ✓ ✗ ✓ 0
OmniAct [21] 9802 ✗ - ✓ ✗ ✓ 0
AgentBench [32] 1091 Multi-isolated ✗ ✗ ✗ ✗ 7
InterCode [57] 1350 (3) Code ✗ ✗ ✗ ✗ 3
MiniWoB++ [30] 125 Web ✗ ✓ ✗ ✗ 125
WebShop [58] 12kk (1) Web ✗ ✓ ✗ ✗ 1
WebArena [66] 812 (241) Web ✗ ✓ ✗ ✗ 5
VWebArena [22] 910 (314) Web ✗ ✓ ✗ ✗ 6
WorkArena [10] 23kk (29) Web ✗ ✓ ✗ ✓ 7
WikiHow [61] 150 (16) Mobile ✗ ✓ ✗ ✗ 16
AssistGUI [13] 100 ✗ ✗ ✓ ✗ ✓ 2
OSWorld 369 Computer ✓ ✓ ✓ ✓ 134
Table 5: Success rates of baseline LLM and VLM agents on OSWorld , grouped by task categories: OS, Office (LibreOffice Calc, Impress, Writer), Daily (Chrome, VLC Player, Thunderbird), Professional (VS Code and GIMP) and Workflow (tasks involving multiple apps), for gaining insights from interfaces and operation logic. See App. C.1 and C.5 for more details.
Inputs Model Success Rate (↑)
OS Office Daily Profess. Workflow Overall
A11y tree Mixtral-8x7B 12.50% 1.01% 4.79% 6.12% 0.09% 2.98%
Llama-3-70B 4.17% 1.87% 2.71% 0.00% 0.93% 1.61%
GPT-3.5 4.17% 4.43% 2.71% 0.00% 1.62% 2.69%
GPT-4 20.83% 3.58% 25.64% 26.53% 2.97% 12.24%
Gemini-Pro 4.17% 1.71% 3.99% 4.08% 0.63% 2.37%
Gemini-Pro-1.5 12.50% 2.56% 7.83% 4.08% 3.60% 4.81%
Qwen-Max 29.17% 3.58% 8.36% 10.20% 2.61% 6.87%
GPT-4o 20.83% 6.99% 16.81% 16.33% 7.56% 11.36%
Screenshot CogAgent 4.17% 0.85% 2.71% 0.00% 0.00% 1.11%
GPT-4V 12.50% 1.86% 7.58% 4.08% 6.04% 5.26%
Gemini-ProV 8.33% 3.58% 6.55% 16.33% 2.08% 5.80%
Gemini-Pro-1.5 12.50% 6.99% 2.71% 6.12% 3.60% 5.40%
Claude-3-Opus 4.17% 1.87% 2.71% 2.04% 2.61% 2.42%
GPT-4o 8.33% 3.58% 6.07% 4.08% 5.58% 5.03%
Screenshot CogAgent 4.17% 0.85% 2.71% 0.62% 0.09% 1.32%
+ A11y tree GPT-4V 16.66% 6.99% 24.50% 18.37% 4.64% 12.17%
Gemini-ProV 4.17% 4.43% 6.55% 0.00% 1.52% 3.48%
Gemini-Pro-1.5 12.50% 3.58% 7.83% 8.16% 1.52% 5.10%
Claude-3-Opus 12.50% 3.57% 5.27% 8.16% 1.00% 4.41%
GPT-4o 41.67% 6.16% 12.33% 14.29% 7.46% 11.21%
Set-of-Mark CogAgent 4.17% 0.00% 2.71% 0.00% 0.53% 0.99%
GPT-4V 8.33% 8.55% 22.84% 14.28% 6.57% 11.77%
Gemini-ProV 4.17% 1.01% 1.42% 0.00% 0.63% 1.06%
Gemini-Pro-1.5 16.67% 5.13% 12.96% 10.20% 3.60% 7.79%
Claude-3-Opus 12.50% 2.72% 14.24% 6.12% 4.49% 6.72%
GPT-4o 20.83% 3.58% 3.99% 2.04% 3.60% 4.59%
Human Performance 75.00% 71.79% 70.51% 73.47% 73.27% 72.36%
Table 6: Success rate (SR) of GPT-4V (SoM) across different types of tasks.
Task Subset % of Total SR (↑)
Easy 28.72% 16.78%
Medium 40.11% 13.12%
Hard 30.17% 4.59%
Infeasible 8.13% 16.67%
Feasible 91.87% 13.34%
Single-App 72.63% 13.74%
Multi-App Workflow 27.37% 6.57%
Table 7: Comparison of model performance and correlation across operating systems.
OS SR (%) Correlation Coefficient
Ubuntu 4.88 0.7
Windows 2.55
Table 8: Action types and parameters defined in action space computer_13 , a variance we created for the potential reinforcement learning research based on our environment.
Action Type Parameters Note
MOVE_TO x, y Move the cursor to the specified position
CLICK button,
x, y,
num_clicks
Click the left button if the button not specified, otherwise click the specified button; click at the current position if x and y are not specified, otherwise click at the specified position
MOUSE_DOWN button Press the left button if the button not specified, otherwise press the specified button
MOUSE_UP button Release the left button if the button not specified, otherwise release the specified button
RIGHT_CLICK x, y Right click at the current position if x and y are not specified, otherwise right click at the specified position
DOUBLE_CLICK x, y Double click at the current position if x and y are not specified, otherwise double click at the specified position
DRAG_TO x, y Drag the cursor to the specified position with the left button pressed
SCROLL dx, dy Scroll the mouse wheel up or down
TYPING text Type the specified text
PRESS key Press the specified key and release it
KEY_DOWN key Press the specified key
KEY_UP key Release the specified key
HOTKEY keys Press the specified key combination
WAIT - Wait until the next action
FAIL - Decide the task cannot be performed
DONE - Decide the task is done
Table 9: Task Example Resources
App Resources Link
OS Ubuntu Documentations https://help.ubuntu.com/
Ask Ubuntu https://askubuntu.com/
Super User https://superuser.com/
Stack Overflow https://stackoverflow.com
YouTube https://www.youtube.com/
Calc LibreOffice Help https://help.libreoffice.org/
Microsoft Tech Community https://techcommunity.microsoft.com/
libreofficehelp.com https://www.libreofficehelp.com/
Reddit r/LibreOfficeCal https://www.reddit.com/r/LibreOfficeCalc/
Reddit r/Excel https://www.reddit.com/r/Excel/
Super User https://superuser.com/
Medium https://medium.com/
Quora https://www.quora.com/
YouTube https://www.youtube.com/
Ryan and Debi & Toren Personal Site https://www.ryananddebi.com/
Writer LibreOffice Help https://help.libreoffice.org/
LibreOffice Forum https://ask.libreoffice.org/
libreofficehelp.com https://www.libreofficehelp.com/
Super User https://superuser.com/
Stack Overflow https://stackoverflow.com
Ask Ubuntu https://askubuntu.com/
Quora https://www.quora.com/
YouTube https://www.youtube.com/
SearchStar Personal Site https://seekstar.github.io/
Impress LibreOffice Help https://help.libreoffice.org/
libreofficehelp.com https://www.libreofficehelp.com/
Reddit r/LibreOffice https://www.reddit.com/r/LibreOffice/
Super User https://superuser.com/
Stack Overflow https://stackoverflow.com
Technical Tips https://technical-tips.com/
Just Click Here https://justclickhere.co.uk/
TikTok https://www.tiktok.com/
VLC VLC Documentation https://docs.videolan.me
VLCHelp.com https://www.vlchelp.com/
VideoLAN’s Wiki https://wiki.videolan.org/
Ubuntu Documentations https://help.ubuntu.com/
Reddit r/Fedora https://www.reddit.com/r/Fedora/
Super User https://superuser.com/
Medium https://medium.com/
YouTube https://www.youtube.com/
Dedoimedo https://www.dedoimedo.com/index.html
Thunderbird Thunderbird Support https://support.mozilla.org/en-US/products/thunderbird
Reddit r/Thunderbird https://www.reddit.com/r/Thunderbird/
Reddit r/Automation https://www.reddit.com/r/automation/
Super User https://superuser.com/
WikiHow https://www.wikihow.com/
Quora https://www.quora.com/
BitRecover https://www.bitrecover.com/
AdSigner https://www.adsigner.com/
Chrome Google Chrome Help https://support.google.com/chrome
Reddit r/Chrome https://www.reddit.com/r/Chrome/
Super User https://superuser.com/
WikiHow https://www.wikihow.com/
in5steps.com https://in5stepstutorials.com/
How-To Geek https://www.howtogeek.com/
Medium https://medium.com/
Quora https://www.quora.com/
YouTube https://www.youtube.com/
Laptop Mag https://www.laptopmag.com
VS Code Super User https://superuser.com/
Stack Overflow https://stackoverflow.com
Quora https://www.quora.com/
YouTube https://www.youtube.com/
Campbell Muscle Lab GitHub https://campbell-muscle-lab.github.io/
GIMP Reddit r/GIMP https://www.reddit.com/r/GIMP/
Super User https://superuser.com/
Stack Overflow https://stackoverflow.com
Quora https://www.quora.com/
Make-Use-Of https://www.makeuseof.com/
YouTube https://www.youtube.com/
Workflow UniPath Marketplace https://marketplace.uipath.com/
sync.blue https://www.sync.blue/
Device Tests https://devicetests.com/
Make Tech Easier https://www.maketecheasier.com/
Unix & Linux Stack Exchange https://unix.stackexchange.com/
Geeks for Geeks https://www.geeksforgeeks.org/
I Love Free Software https://www.ilovefreesoftware.com/
The Geek Diary https://www.thegeekdiary.com/
Zyxware https://www.zyxware.com/
GNOME Discourse https://discourse.gnome.org/
It’s FOSS https://itsfoss.com/
Super User https://superuser.com/
Stack Overflow https://stackoverflow.com
LibreOffice Forum https://ask.libreoffice.org/
ImpressExtractNotes https://github.com/danielrcollins1/ImpressExtractNotes
Medium https://medium.com/
YouTube https://www.youtube.com/
Kelvin Smith Library https://case.edu/library/
Table 10: Detailed statistics of OSWorld benchmark suite about examples number, average instruction tokens, infeasible instructions and integrated instructions.
OS Calc Impress Writer VLC Thunderbird Chrome VSCode GIMP Workflow Overall
Examples 24 47 47 23 17 15 46 23 26 101 369
Avg. Inst. Tokens 22.38 33.30 25.19 35.30 35.82 34.07 22.07 20.78 16.23 51.24 33.36
#Infeasible 5 1 0 1 3 1 3 5 10 1 30
#Integrated 7 19 30 0 0 0 26 0 0 2 84
Table 11: Detailed statistics of Windows analytic set benchmark suite. This set contains no infeasible tasks and integrated tasks.
Excel Word PPT Workflow Overall
Examples 11 9 7 16 43
Avg. Inst. Tokens 19.45 21.44 21.86 47.57 32.48
Table 12: More Example Showcase from Each Subset of Domains.
Related App(s) Instruction(s) Screenshot Abilities Needed
OS I want to install Spotify on my current system. Could you please help me? [Uncaptioned image] knowledge of OS; omit distractions
Calc Check the names in column “Names with duplicates” and put the unique ones in column “Unique Names”. Keep the original order. [Uncaptioned image] massive elements; knowledge tricks or reasoning over long actions
Calc I have a lookup table for the officers of each branch. Please, here is another table in which I need to fill with the officer names according the headoffice (i.e., the branch name). Help me to complete this. [Uncaptioned image] massive elements; knowledge of formulas and functions
Calc Given a partial calendar, please highlight all the weekends (Saturday & Sunday) by setting the cell background as red (#ff0000). [Uncaptioned image] massive elements; commonsense reasoning; software tricks
Impress I closed the slide pannel on the left and idk how to get it back please help [Uncaptioned image] software knowledge; imagine about UI layouts; overcome typos in instruction
Impress On it Whenever I launch a LibreOffice Impress, it uses both screens, one for current slide and next slide and another for actual presentation. What I want is to use only one monitor which shows presentation. I dont want the screen with Current slide and Next slide so that it can be used for other purposes. How should I achieve this? [Uncaptioned image] reason from unprofessional phenomenon expression
Writer Copy the screenshot 1.png from the desktop to where my cursor is located [Uncaptioned image] locate the position of cursor; switch from desktop and app
Chrome Can you help me clean up my computer by getting rid of all the tracking things that Amazon might have saved? I want to make sure my browsing is private and those sites don’t remember me. [Uncaptioned image] understanding the unprofessional expression
VLC I am reading lecture note in PDF while a music video is running in VLC media player. But I find I need to switch to the player every time I need to pause/start.Could you help me change the setting to allow pausing the video using keyboard shortcut without minimizing the PDF reader? I want to focus on the lecture note and don’t be disturbed by the app switching. [Uncaptioned image] understanding the reference from unprofessional expression; software knowledge
VLC Hey, could you turn this video the right way up for me? And once it’s flipped around, could you save it for me with the name ‘1984_Apple.mp4’ on the main screen where all my files are? [Uncaptioned image] software knowledge; spatial judgment ability
Thunderbird Create a local folder called "Promotions" and create a filter to auto move the inbox emails whose subject contains “discount” to the new folder [Uncaptioned image] software knowledge
Thunderbird Attach the my AWS bill to the email. The bill is stored at  /aws-bill.pdf. Don’t close it or send it. I haven’t finish all the contents. [Uncaptioned image] file management; extra requirement
Thunderbird I’ve got a bunch of email accounts in Thunderbird, and it’s a hassle to check them one by one. Can you show me how to set up a unified inbox so I can see all my emails in one place? [Uncaptioned image] deep-hided feature, need to be explored even by human users; pop-up window
VS Code Please modify VS Code’s settings to disable error reporting for Python missing imports. [Uncaptioned image] software knowledge to deal with settings; reasoning to understand the cause and solution of the error
VS Code Please help me install the autoDocstring extension in VS Code. [Uncaptioned image] software knowledge to deal with Extensions; reasoning to search and install the extension successfully
GIMP Could you make the background of this image transparent for me? [Uncaptioned image] precise and intricate operations
GIMP Help me choose the yellow triangle and position it at the center of my picture. [Uncaptioned image] spatial perception and reasoning, as well as precise control of actions
Multiple (VLC+GIMP) Could you help me create an Animated GIF from a video file using VLC and GIMP from the source of video “src.mp4”, 5-second clip beginning at 00:03? [Uncaptioned image] software knowledge to undergo sophisticated processes and ability to process multi-step procedure successfully
Multiple (ThunderBird+ Writer+Chrome) Help me export charts, graph or other images from docx files received in email “Lecture Document” in Notes folder and upload these png files to the figures/ folder in Google Drive for later use (use numbers to name them). [Uncaptioned image] ability to selectively export charts, graphs and images from docx file; software knowledge for google drive file upload
Multiple (Chrome+Calc) Could you help me extract data in the table from a new invoice uploaded to my Google Drive, then export it to a Libreoffice calc .xlsx file in the desktop? [Uncaptioned image] ability to do table data extraction; export data to .xlsx file
Table 13: Criteria for keeping a11y tree nodes on Ubuntu and Windows platforms
Condition Ubuntu Windows
Node Tags document∗*, ∗*item, ∗*button, ∗*heading, ∗*label, ∗*scrollbar, ∗*searchbox, ∗*textbox, ∗*link, ∗*tabelement, ∗*textfield, ∗*textarea, ∗*menu, alert, canvas, check-box, combo-box, entry, icon, image, paragraph, scroll-bar, section, slider, static, table-cell, terminal, text, netuiribbontab, start, trayclockwclass, traydummysearchcontrol, uiimage, uiproperty, uiribboncommandbar Same as Ubuntu
Showing True Not Applicable
Visible True True
Enabled or Editable or Expandable or Checkable True True
Has Name or Text or Image True True
Position >=0>=0 >=0>=0
Size >0>0 >0>0
Table 14: Detailed success rates of baseline LLM and VLM agents on OSWorld , divided by apps (domains): OS, LibreOffice Calc, LibreOffice Impress, LibreOffice Writer, Chrome, VLC Player, Thunderbird, VS Code, GIMP and Workflow which is comprehensive with multiple apps, for gaining insights from interfaces and operation logics.
Inputs Model Success Rate (↑)
OS Calc Impress Writer VLC TB Chrome VSC GIMP Workflow
A11y Mixtral-8x7B 12.50 0.00 0.39 4.34 10.22 6.67 2.17 8.69 3.85 0.10
GPT-3.5 4.17 2.13 6.77 4.35 6.53 0.00 2.17 0.00 0.00 1.62
Gemini-Pro 4.17 0.00 2.13 4.35 12.41 0.00 2.17 0.00 7.69 0.63
GPT-4 20.83 0.00 6.77 4.35 23.53 26.67 26.09 30.43 23.08 2.97
Gemini-Pro-1.5 12.50 2.13 2.13 4.35 6.53 0.00 10.87 8.70 0.00 3.60
Llama-3-70B 4.17 0.00 0.39 8.70 6.53 0.00 2.17 0.00 0.00 0.63
GPT-4o 20.83 6.38 6.77 8.69 12.41 20.00 17.39 21.74 11.54 7.56
Qwen-Max 29.17 0.00 2.52 13.04 8.95 0.00 10.87 8.70 11.54 2.61
Screen CogAgent 4.17 0.00 0.00 4.34 6.53 0.00 2.17 0.00 0.00 0.00
Gemini-ProV 8.33 0.00 6.77 4.35 12.41 0.00 6.52 8.70 23.08 2.08
GPT-4V 12.50 0.00 2.52 4.35 18.34 0.00 6.52 0.00 7.69 6.04
Claude-3-Opus 4.17 0.00 2.52 4.34 6.53 0.00 2.17 0.00 3.84 2.61
Gemini-Pro-1.5 12.50 0.00 13.16 8.70 6.53 0.00 2.17 0.00 11.54 3.60
GPT-4o 8.33 0.00 6.77 4.35 16.10 0.00 4.35 4.35 3.85 5.58
Screen CogAgent 4.17 2.17 0.00 4.35 6.53 0.00 2.17 0.00 0.00 0.10
+ A11y Gemini-ProV 4.17 2.13 6.77 4.35 18.30 0.00 4.35 0.00 0.00 1.52
GPT-4V 16.67 0.00 6.77 21.73 24.18 33.33 21.74 21.74 15.38 4.59
Claude-3-Opus 12.50 2.13 4.65 4.34 18.30 0.00 2.17 8.69 7.69 0.99
Gemini-Pro-1.5 12.50 0.00 4.65 8.70 12.41 0.00 8.70 4.35 11.54 1.56
GPT-4o 41.67 4.26 6.81 8.70 9.50 6.67 15.22 30.43 0.00 7.46
SoM CogAgent 4.17 2.17 0.00 4.34 6.53 0.00 2.17 0.00 0.00 0.00
Gemini-ProV 4.17 0.00 0.39 4.34 6.53 0.00 0.00 0.00 0.00 0.63
GPT-4V 8.33 8.51 6.38 13.04 10.66 33.33 23.91 17.39 11.54 6.57
Claude-3-Opus 12.50 2.13 0.39 8.70 6.53 13.33 17.39 0.00 11.54 4.49
Gemini-Pro-1.5 16.67 0.00 10.64 4.35 24.18 13.33 8.70 4.35 15.38 3.60
GPT-4o 20.83 0.00 6.77 4.35 6.53 0.00 4.35 4.35 0.00 3.60
Human Performance 75.00 61.70 80.85 73.91 70.59 46.67 78.26 73.91 73.08 73.27