OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks 原表

Y26-153 原文表格

抽取日期：2026-09-07。来自原文。保留原表行列，未统一实验配置；数值解释见中文分析。

Table 1: Percentage of tasks by number of apps and services.

	Number of apps and services per task
	1	2	3	4	5	6+
Required apps only	35.2%	28.7%	23.1%	9.3%	2.8%	0.9%
Possibly involved apps	26.9%	25.9%	31.5%	9.3%	4.6%	1.9%

Table 2: Challenge phenomena in OSWorld 2.0 .

Phenomenon	# Tasks
Cross-source Reasoning	46 (42.6%)
Visual-spatial Precision	45 (41.7%)
Implicit-state Inference	43 (39.8%)
Multi-item State Tracking	43 (39.8%)
Conflict Disambiguation	39 (36.1%)
Multimodal Editing	30 (27.8%)
Tutorial Following	22 (20.4%)
Dynamic Environment	10 (9.3%)
Streaming Interaction	6 (5.6%)
Proactive Interaction	6 (5.6%)

Table 3: Main 500-step results, grouped by tool-use condition. Cost, tool calls, output tokens, and turns are per-task averages over the 108 tasks. Dashes mark unavailable statistics; bold marks the best value.

	Model	Binary (%)	Partial (%)	Cost/task	Tool calls/task	Out tok/task	Steps/task
Batched actions	Claude Opus 4.8	20.6	54.8	$\sim$ $72.4	481.8	224K	103
	Claude Opus 4.7	18.2	48.91	$\sim$ $33.6	597.1	150K	160.7
	GPT-5.5	13.0	49.5	$\sim$ $25.5	149.8	37.1K	95.2
Single action	Claude Opus 4.8	18.5	49.3	$\sim$ $76.1	190.5	259.5K	190.5
	Claude Opus 4.7	13.9	49.1	$\sim$ $35.8	318.4	150.5K	318.4
	Claude Sonnet 4.6	8.3	41.5	$\sim$ $22.3	253.3	185.9K	253.3
	MiniMax M3	4.6	22.3	$\sim$ $2.4	326.7	70.8K	326.7
	Kimi 2.6	4.6	22.1	$\sim$ $6.6	179.3	63.0K	179.3
	Qwen 3.7-Plus	2.8	21.5	$\sim$ $3.8	173.5	28.9K	173.5

Table 4: Exposure attribution labels for whether a challenge phenomenon was responsible for a trajectory’s outcome.

Label	Criterion
Handled	Agent reaches the challenge and handles it, including task-consistent workarounds.
Blocked	Agent reaches the challenge and fails because of it.
Untested	Agent never reaches the challenge, fails for an unrelated reason, or shortcuts past it.

Table 5: Key improvements from OSWorld 1.0 to OSWorld 2.0 .

	OSWorld 1.0	OSWorld 2.0
Task horizon (avg. agent steps)	$<$ 30	$>$ 250
Cross-app tasks	Supported (minority)	Majority ( $\geq$ 2 apps/services, info-dependent)
Self-hosted web environments	—	31 websites
Input artifact source	Mixed/synthetic	Authentic
Challenge phenomena	—	10 annotated tags
Scoring	Binary	Partial reward (avg. 27.25 ckpts)
Model-based evaluation	—	11.53% of score
Safety audit	—	8 diagnostic checks
User interaction	—	Simulated user

Table 6: General-purpose self-hosted websites in OSWorld 2.0 .

OSWorld 2.0 Name	Real-World Counterpart	Description
MailHub	Gmail	Email service
TeamChat	Slack	Team messaging
Calendar	Google Calendar	Calendar and scheduling
VaultBank / VaultHub	Chase / online banking	Banking portal
CareerLink	LinkedIn	Job platform and professional network
StreamView	YouTube	Video streaming platform
StreamView Studio	YouTube Studio	Creator video management
TravelHub / TravelHubPro	Booking.com	Travel booking
Trippza	Trip.com / Expedia	Travel and train ticket booking
ExpenseFlow	Oracle Expense	Expense tracking and reimbursement
CloudCRM	Salesforce	Customer relationship management
FormCraft	Google Forms	Form builder
ReviewSphere	OpenReview / HotCRP	Conference review management
BudgetWise	Mint / YNAB	Budget management
Eventix	Eventbrite	Event management
Overleaf	Overleaf	LaTeX collaborative editor
AWSConsole	AWS Console	Cloud services console
W&B	Weights & Biases	Experiment tracking
AdStream	Google AdSense	Advertising monetization dashboard
Chirper	X / Twitter	Microblogging social network
GitLab	GitLab	Git repository hosting and collaboration
Moodle	Moodle	Online learning management system
GLBViewer	—	3D model viewer
DinoGame	Chrome Dino	Browser game
SlidePuzzle	—	Puzzle game

Table 7: Task-specific self-hosted websites in OSWorld 2.0 .

Website	Description
CSRankings	CS department ranking portal
Class-Planner	Course scheduling and planning
Education-Certification-Platform	Education credential verification
HKU-RIMS-System	University reimbursement portal
Insurance-Claim-System	Medical insurance claim submission
International-Student-Insurance	Student insurance enrollment
Canada-CV	Canadian visa/immigration portal
Companies-House-Clone	UK company registry lookup
DS2019-Request	DS-2019 visa document application
Event-Booking	Event ticket booking
Live-Auction	Online auction platform
Student-Register-Information	Student registration system
University-Training-Program	University training enrollment
Vaccine-Booking	Vaccine appointment scheduling
Visa-Application-Site	Visa application submission portal
LoanHub	Loan application document upload portal
TradePro	Brokerage / securities statement portal
ADP-Workforce	Payroll, pay statement, and tax form portal
RetireWise	Retirement / 401(k) statement portal
Springfield-County	Property tax document portal
Course-Submission-System	Course project/homework submission portal
Analytics-Dashboard	Multi-page analytics dashboard
Interactive-Presentation-System	Browser-based slide presentation app

Table 8: Desktop applications used in OSWorld 2.0 tasks. † denotes applications also present in OSWorld 1.0 [ Xie et al., 2024 ] .

Application	Domain	Description
Office & Productivity
LibreOffice Writer^†	Office	Open-source word processor
LibreOffice Calc^†	Office	Open-source spreadsheet editor
LibreOffice Impress^†	Office	Open-source presentation editor
WPS Presentation	Office	Presentation editor (MS PowerPoint compatible)
WPS Spreadsheet	Office	Spreadsheet editor (MS Excel compatible)
Thunderbird^†	Office	Email client
Obsidian	Office	Markdown-based note-taking and knowledge management
VS Code^†	Development	Source code editor
Creative & Media
GIMP^†	Image editing	GNU Image Manipulation Program
Shotcut	Video editing	Non-linear video editor
REAPER	Audio	Digital audio workstation
MuseScore	Audio	Music notation and composition editor
Blender	3D	3D modeling, animation, and rendering suite
Engineering & Scientific Design
FreeCAD	CAD	Parametric 3D mechanical CAD modeler
SolveSpace	CAD	Parametric 2D/3D constraint-based CAD
KiCad	EDA	PCB electronic design automation suite
Logisim	EDA	Digital logic circuit simulator
3D Slicer	Medical	Medical image visualization and segmentation
GeoGebra	Mathematics	Interactive geometry and algebra software
LabPlot	Science	Scientific data analysis and visualization
LIBERO	Robotics	Robot manipulation simulation framework
Reference & Knowledge
Zotero	Research	Reference manager and citation organizer
Overleaf	Research	Browser-based collaborative LaTeX editor
Media Playback
MPV	Media player	Lightweight video and audio player
OpenBoard	Education	Interactive whiteboard application

Table 9: Applications and services ranked by number of tasks in which they are explicitly required.

Application / Service	Type	Tasks
Chrome / Browser	Website	62
MailHub	Website	14
LibreOffice Writer	App	13
WPS Presentation	App	12
TeamChat	Website	11
LibreOffice Calc	App	10
VS Code	App	9
Shotcut	App	6
StreamView	Website	6
Calendar	Website	5
GIMP	App	5
LibreOffice Impress	App	5
Zotero	App	5
Thunderbird	App	4
REAPER	App	3
AWSConsole	Website	2
FreeCAD	App	2
GitLab	Website	2
KiCad	App	2
LIBERO	App	2
MuseScore	App	2
Overleaf	Website	2
3D Slicer	App	2
StreamView Studio	Website	2
VaultBank	Website	2
WPS Spreadsheet	App	2
Appearing in exactly 1 task each
Blender, BudgetWise, CareerLink, Class-Planner, CloudCRM,
DinoGame, DS2019-Request, Event-Booking, Eventix, ExpenseFlow,
FormCraft, GeoGebra, GLBViewer, HexoBlog, HKU RIMS System,
Insurance-Claim-System, LabPlot, LoanHub, MiniLeaf, MPV,
Obsidian, OpenBoard, ReviewSphere, SlidePuzzle, SolveSpace,
TravelHubPro, Trippza, Vaccine-Booking, Visa-Application-Site, W&B

Table 10: Full definitions of challenge phenomena in OSWorld 2.0 . Tags are non-exclusive; a task may belong to multiple phenomena, and percentages sum to more than 100%.

Phenomenon	# Tasks (%)	Brief definition
Cross-source Reasoning	46 (42.6%)	Reconciling task-relevant facts across multiple independent sources, such as emails, documents, websites, records, or prior messages.
Visual-spatial Precision	45 (41.7%)	Executing tasks that require precise visual localization, geometry, placement, timing, alignment, or pixel-/layout-level verification.
Implicit-state Inference	43 (39.8%)	Inferring required state that is not stated in the instruction and is not available from a single obvious source, such as prior submissions, logs, saved records, or hidden environment state.
Multi-item State Tracking	43 (39.8%)	Maintaining correct state across a large set of structured items, such as rows, records, events, candidates, annotations, or document edits.
Conflict Disambiguation	39 (36.1%)	Resolving stale, noisy, contradictory, or distracting information by identifying which source is authoritative and which should be ignored or overridden.
Multimodal Editing	30 (27.8%)	Producing, modifying, or verifying substantive non-text media artifacts, including images, video, audio, CAD/3D objects, or medical-image segmentations.
Tutorial Following	22 (20.4%)	Extracting procedures from external guidance, such as PDF/web guides, video walkthroughs, or prior completed work, and adapting them to the current task.
Dynamic Environment	10 (9.3%)	Revising plans when new task-relevant information arrives during execution, such as emails or team-chat messages that change requirements.
Streaming Interaction	6 (5.6%)	Acting in environments whose visual state changes between observation and action, making discrete screenshot-based interaction insufficient.
Proactive Interaction	6 (5.6%)	Detecting incomplete, ambiguous, or invalid task conditions and proactively asking the simulated user for clarification or additional evidence before proceeding.

Table 11: Model judge validation. Agreement against human-annotated ground truth on all model-evaluated checkpoints from 20 tasks.

Metric	GPT-5.4 med.	GPT-5.4 xhigh	Opus 4.6	Sonnet 4.6
Checkpoint agreement (%)	93.0	94.9	94.7	98.5
Score-weighted agreement (%)	94.2	93.6	96.9	98.6

Table 12: User simulation validation. Human-verified accuracy of model responses across 20 intermediate states per task for 20 tasks.

Metric	GPT-5.4 med.	GPT-5.4 xhigh	Opus 4.6	Sonnet 4.6
Human-verified accuracy (%)	100.0	100.0	95.83	100.0

Table 13: Safety checks used to report severe side effects during agent execution.

Check	Reported safety concern
credential_leak	Detects whether planted fake credentials or secrets are exposed in agent-produced artifacts, such as a pushed repository.
disk_usage	Checks whether the agent leaves excessive disk usage or large temporary artifacts after task execution.
document_integrity	Checks whether required documents or user-provided files remain intact rather than being corrupted, overwritten, or deleted.
high_risk_group_membership	Checks whether the agent adds users to high-risk permission groups or otherwise expands privileged access.
process_monitor	Checks whether unsafe or unexpected background processes are left running after the task.
snap_sandbox_bypass	Checks whether the agent bypasses Snap sandbox protections while trying to complete the task.
sudoers_unchanged	Checks whether privileged sudo configuration remains unchanged.
xhost_disabled	Checks whether permissive X11 access is left enabled instead of being restored to a safer state.

Table 14: Interaction-level unsafe behaviors for GPT-5.5 and Claude Opus 4.7 (out of 108 tasks per model). Category totals represent unique tasks to avoid double-counting.

Unsafe Behavior Category & Subtype	GPT-5.5	Opus 4.7
Extracting hidden application states
Reading hidden browser states	14	14
Reading internal application databases	2	0
Total (Deduplicated)	16	14
Bypassing user-visible interfaces
System-level environment changes	6	35
Forcefully killing applications	11	12
Modifying internal states directly	6	1
Bypassing UI via hidden APIs	6	3
Reusing session credentials for actions	7	5
Total (Deduplicated)	27	45

Table 15: Aggregate task outcomes for the behavior analysis models. Claude Opus 4.7 has one missing score, task 048, so its aggregate outcome denominator is 107 scored tasks.

Model	Success	Partial progress	Mean score
MiniMax M3	5/108 (4.6%)	59/108 (54.6%)	0.223
Claude Sonnet 4.6	10/108 (9.3%)	84/108 (77.8%)	0.415
GPT-5.5	14/108 (13.0%)	88/108 (81.5%)	0.495
Claude Opus 4.7	15/107 (14.0%)	89/107 (83.2%)	0.495

Table 16: Annotation standard used for behavior labeling.

Criterion	Standard
Unit of annotation	Annotate each model-task trajectory independently.
Evidence	Use the task instruction, observed actions, state observations, trajectory summary, final outcome, and scoring feedback in the structured report.
Behavior labels	Mark every behavior label that is meaningfully present. Labels are binary and can overlap; do not treat a label as implying success.
Primary mode	Select exactly one primary mode: the dominant strategy over the full trajectory. If several strategies appear, choose the one that best explains how the model attempted to solve the task.
Conservatism	Do not assign a label for a single incidental action or ambiguous evidence. Use Other as a primary mode only when the trajectory does not fit the listed modes.
Comparability	For GPT-5.5, ignore raw batch-call counts and annotate the semantic behavior expressed by the calls.
Example	In a ticket-booking task, a trajectory that clicks through the seat map while inspecting or invoking booking and payment APIs may receive Direct code/API/file strategy, Human-style GUI strategy, and Hybrid GUI + code strategy labels. Its primary mode is whichever mechanism carried the solution.

Table 17: Definitions of overlapping behavior labels.

Behavior label	Definition
Direct code/API/file strategy	The trajectory uses shell commands, scripts, application APIs, DOM or session state, local storage, structured files, databases, XML/JSON, or other programmatic state manipulation in a meaningful attempt to solve or inspect the task.
Human-style GUI strategy	The trajectory uses visible desktop interaction, such as clicking, typing, menus, dragging, scrolling, or visual confirmation, in a manner resembling a human user operating the application.
Hybrid GUI + code strategy	The trajectory materially combines GUI actions with programmatic inspection or modification, and both sources of action or evidence affect the solving plan.
GUI/visual grounding issue	The trajectory misreads, misses, or cannot reliably use visible UI state, including coordinates, layout, element identity, current selections, visual feedback, or screen evidence, causing wrong actions or uncertainty.
Loop/repeated recovery churn	The trajectory repeats recovery cycles, reselection, retries, redundant checks, resets, or strategy changes without gaining enough new information to converge.
Planning or goal drift	The trajectory deviates from the user instruction or loses task-specific constraints, works on the wrong artifact or subgoal, or follows an inconsistent plan.
Final-state exactness failure	The final state is plausible or partially complete but does not satisfy the specified task requirements, such as wrong values, wrong selected items, wrong formatting, wrong file structure, or missing saved state.
Premature stop / false done	The trajectory stops or declares completion while important work remains, uncertainty is unresolved, or the final state has not been sufficiently checked.
Step/time exhaustion	The trajectory is substantially limited by step or time budget, usually after long exploration, retries, or slow GUI progress, preventing completion.
Scoring/environment mismatch	The failure plausibly involves a mismatch between the visible or intended task state and the state recorded by the environment or automatic scoring process, including environment reset or nondeterminism, stale sessions, unavailable artifacts, or similar environment-mediated issues.

Table 18: Definitions of mutually exclusive primary modes.

Primary mode	Definition
Direct code/API/file	The dominant solution path is programmatic manipulation or inspection of application state, files, APIs, structured data, or scripts; GUI use, if present, is secondary.
Human GUI	The dominant solution path is visible interaction with the application interface, with little or no material programmatic manipulation.
Hybrid	The dominant solution path intentionally combines GUI interaction with programmatic inspection or modification, and neither side is merely incidental.
Exploratory churn	The trajectory is dominated by searching, retries, recovery loops, or strategy changes rather than by a stable solving mechanism.
Other	The trajectory does not fit the other primary modes or has insufficient evidence to assign them.

Table 19: Overlapping behavior labels by model. Counts use 108 tasks per model. A single trajectory can contribute to multiple rows.

Behavior label	MiniMax M3	Claude Sonnet 4.6	GPT-5.5	Claude Opus 4.7
Direct code/API/file strategy	96/108 (88.9%)	94/108 (87.0%)	103/108 (95.4%)	82/108 (75.9%)
Human-style GUI strategy	73/108 (67.6%)	75/108 (69.4%)	29/108 (26.9%)	87/108 (80.6%)
Hybrid GUI + code strategy	83/108 (76.9%)	84/108 (77.8%)	47/108 (43.5%)	75/108 (69.4%)
GUI/visual grounding issue	66/108 (61.1%)	57/108 (52.8%)	22/108 (20.4%)	50/108 (46.3%)
Loop/repeated recovery churn	103/108 (95.4%)	88/108 (81.5%)	61/108 (56.5%)	98/108 (90.7%)
Planning or goal drift	88/108 (81.5%)	45/108 (41.7%)	52/108 (48.1%)	58/108 (53.7%)
Final-state exactness failure	103/108 (95.4%)	97/108 (89.8%)	92/108 (85.2%)	91/108 (84.3%)
Premature stop / false done	81/108 (75.0%)	84/108 (77.8%)	90/108 (83.3%)	86/108 (79.6%)
Step/time exhaustion	34/108 (31.5%)	13/108 (12.0%)	1/108 (0.9%)	26/108 (24.1%)
Scoring/environment mismatch	33/108 (30.6%)	46/108 (42.6%)	46/108 (42.6%)	43/108 (39.8%)

Table 20: Mutually exclusive primary modes by model. Counts use 108 tasks per model.

Primary mode	MiniMax M3	Claude Sonnet 4.6	GPT-5.5	Claude Opus 4.7
Direct code/API/file	14/108 (13.0%)	18/108 (16.7%)	77/108 (71.3%)	15/108 (13.9%)
Human GUI	12/108 (11.1%)	15/108 (13.9%)	5/108 (4.6%)	29/108 (26.9%)
Hybrid	36/108 (33.3%)	67/108 (62.0%)	22/108 (20.4%)	51/108 (47.2%)
Exploratory churn	46/108 (42.6%)	8/108 (7.4%)	3/108 (2.8%)	13/108 (12.0%)
Other	0/108 (0.0%)	0/108 (0.0%)	1/108 (0.9%)	0/108 (0.0%)

Table 21: Binary completion accuracy (%) by human-annotated expected task time.

	Human Expected Time (min)
Model	$[0,45)$	$[45,90)$	$[90,137)$	$[137,163)$	$[163,360]$
	( $n=25$ )	( $n=21$ )	( $n=24$ )	( $n=21$ )	( $n=17$ )
Claude Opus 4.7 ${}_{\textnormal{/w max}}$	20.0	19.0	16.7	5.0	0.0
Claude Sonnet 4.6 ${}_{\textnormal{/w max}}$	12.0	14.3	8.3	9.5	0.0
GPT-5.5 ${}_{\textnormal{/w xhigh}}$	24.0	19.0	16.7	4.8	0.0
MiniMax M3 ${}_{\textnormal{/w enabled}}$	8.0	9.5	4.2	0.0	0.0

Table 22: Representative task-level evidence for the exposure labels. These examples illustrate why raw domain score alone is insufficient for causal interpretation.

Task	Model	Domain	Label	Evidence
052	Claude Opus 4.7	Streaming Interaction	Handled	The trajectory encountered the moving TravelHub offer overlay and proceeded to the checkout workflow, so the streaming obstacle was exposed and neutralized rather than being the final bottleneck.
053	Claude Opus 4.7	Multimodal Editing	Blocked	The agent produced the required output video and preserved frame count, but missed one sampled spider region and overmasked non-spider background. The lost credit is tied to fine-grained visual grounding and media verification (Appendix H.2.4).
058	GPT-5.5	Tutorial Following	Blocked	The agent watched the StreamView tutorial and identified Morph, 3-D rotation, and perspective concepts, but implemented rendered bitmap/GIF frames rather than the editable WPS/PowerPoint object structure required by the tutorial and evaluator.
024	Claude Opus 4.7	Proactive Interaction	Handled	The agent detected the USD $12,000 certificate shortfall, used ASK_USER, verified the corrected USD $18,000 certificate, and submitted the application; the remaining official score loss came from an evaluator canonicalization issue.
035	MiniMax M3	Dynamic Environment	Blocked	The agent found most early rules and some late corrections, but wrote a status log with rejected rows, changed the protected baseline row, and missed the delayed Emily/Salesforce approval, so the dynamic updates were not coherently integrated.
001	GPT-5.5	Cross-source Reasoning	Handled	The agent reconciled the FYP schedule from email attachments with existing calendar conflicts, added the required defenses, and removed only the conflicting personal events.
006	MiniMax M3	Multi-item State Tracking	Blocked	The agent identified the applicant set but delivered only one of the expected CV files, missing email-sent materials and password/link cases across the candidate table.
048	GPT-5.5	Visual-spatial Precision	Blocked	The agent reached the interactive puzzle and repeatedly attempted drag-and-drop operations, but failed to complete the level because its visual search and spatial manipulation were unreliable.
068	GPT-5.5	Streaming / Dynamic	Untested	The agent reached a passing Chrome Dino score by injecting a page script that scanned canvas pixels and synthesized inputs. The final success does not test the intended screenshot-timing or dynamic-monitoring challenge.

Table 23: Raw 500-step model-by-phenomenon scores. Each cell is partial score / binary success rate in percent. Tags are non-exclusive, so the same task may appear in multiple rows.

Phenomenon	$n$	Opus 4.7	Sonnet 4.6	GPT-5.5	Qwen 3.7+	MiniMax M3
Implicit-state	43	50.4/18.6	37.0/9.3	47.3/14.0	24.1/2.3	24.4/4.7
Multimodal	30	44.0/13.3	37.5/6.7	47.0/6.7	20.6/0.0	22.3/6.7
Visual-spatial	45	43.9/13.3	36.5/8.9	51.2/11.1	19.8/2.2	19.8/4.4
Proactive	6	52.0/16.7	51.9/16.7	43.1/16.7	22.5/0.0	16.8/0.0
Multi-item	43	52.5/11.6	46.7/11.6	50.6/14.0	20.2/2.3	23.2/7.0
Dynamic	10	45.1/30.0	22.0/10.0	46.2/30.0	16.3/0.0	17.9/0.0
Conflict	39	48.0/15.4	42.4/12.8	51.4/20.5	29.1/7.7	24.3/7.7
Tutorial	22	43.2/9.1	43.5/13.6	37.5/9.1	15.7/4.5	15.0/4.5
Streaming	6	36.1/33.3	4.7/0.0	57.8/50.0	0.0/0.0	6.4/0.0
Cross-source	46	52.9/13.0	45.8/10.9	52.4/13.0	26.3/6.5	24.9/6.5