OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
OSWorld supplies real computer environments with task initialization and execution-based checks for open-ended desktop and web workflows.
Problem
A computer assistant may need to move among applications, manipulate files, and continue work from a partially prepared state. Evaluating only a predicted action against a recorded click cannot establish whether that work was completed. OSWorld addresses the infrastructure and task-specific validation needed to test such interactions on functioning computers.
Contributions
A computer environment connects an agent to virtual machines running real applications, with support for different observation and action configurations.
Tasks include explicit initial-state preparation and custom evaluators that inspect the files, application settings, or other state relevant to the user's request.
The benchmark spans individual applications, file operations, and multi-application workflows, with human evaluation and diagnostic studies of agent observations and errors.
Method
A task configuration specifies the instruction, required files and application state, and the checks that define completion. The task manager initializes that state before interaction begins. This allows an episode to start from realistic work in progress rather than only from an empty desktop.
The agent observes the machine through screenshots, accessibility information, or a selected combination. A controller executes mouse and keyboard operations inside the virtual machine and returns the resulting observation. The infrastructure supports isolated environments running in parallel.
After the episode, the evaluator can post-process application state, retrieve relevant files or information, and apply a task-specific comparison. For example, a document-editing task requires checking the edited artifact rather than accepting the agent's declaration that it has finished. Some checks use application configuration or external information, so the validation logic varies with the task.
The environment, tasks, and baseline agents are separate components. This permits a new agent to use the same task setup and validator, or a new task to supply its own setup and evaluation functions while reusing the execution machinery.
Evaluation
The original paper evaluates computer tasks involving desktop software, browser applications, file handling, and cross-application workflows. Completion is determined by the supplied execution-based evaluator; human runs provide an additional reference for task feasibility and difficulty.
Experiments vary screenshot and accessibility inputs, resolution, history length, and interface layout. They examine application categories, workflow complexity, and error types. This note describes the original paper's protocol; later OSWorld-Verified and OSWorld-v2 releases have distinct evaluation configurations.
详细阅读笔记 · SUP-A13
本笔记保留阅读时的论文版本、来源与实验边界。文中“你的方案／当前方案”等比较反映当时的讨论；当前研究方向见分类页。自拟例子与复现建议不是作者实验结果。
原论文 · P0 · 真实桌面在线任务。核查日期：2026-09-07。
0. 摘要
中文转述；本卡核查原始OSWorld论文v2（2024-05-30），不是OSWorld2.0，也不是当前排行榜。OSWorld让模型在真实桌面应用中执行开放任务，使用可重置虚拟机与定制终态评估。原论文的主体为Ubuntu上369题，另有43个Windows测试例子。
1. 方法动机
网页或单应用任务不能代表完整桌面工作；离线点击匹配也不能承认多条正确操作路线。作者把文件、浏览器、表格、邮件和专业软件放在共同环境里，检验最终工作结果。这适合检验历史是否支持跨应用信息传递，但成功失败还受定位、应用知识与步数限制影响，不能都归因于记忆。
2. 原文方法与执行过程
§2.1–2.2：先设置初态，再实际行动
任务由自然语言要求、初始环境配置和检查程序组成。虚拟机快照恢复环境，再下载或放置文件、启动软件、调整窗口等。代理根据当前观察生成动作，动作改变实际系统，再获取新观察，直到DONE、FAIL或步数用尽。
§2.2.3：对每类结果设计检查
表格任务可以读取工作簿检查工作表名和单元格；浏览器设置任务可以检查配置或cookie；某些任务检查界面树。实时网页值可能通过动态getter读取。这样的评估比“看起来点对了”更接近任务完成，但仍受评估器覆盖与实时内容变化影响。
自拟例子：从表格读取未缴费邮箱，填入邮件收件人。历史模块必须保存准确字符串；最终检查收件人字段，而非强迫模型按人工同样点击次数操作。若此前读错邮箱，后面正确打开邮件也不能算成功。
§2.3–2.4：观察与动作权限必须明确
原文比较截图、可访问性树、二者结合、SoM四种观察。动作使用pyautogui风格鼠标键盘代码，可组合操作，另有WAIT/FAIL/DONE。环境支持多个操作系统，不代表主体369题在每个系统都完整验证。
§3：组织真实任务与不可完成任务
369题中101个跨应用、268个单应用，30个标为不可完成；这些类别有不同统计轴，不能简单全部相加。正确识别不可完成任务也可能得到奖励。数据有302个初态与134种检查脚本，数字不是模型调用次数。
3. 基线输入与训练
§4.1使用现有LLM/VLM提示式执行，最近3轮观察/动作按对话组织；超过模型限制时从早期截断。原论文默认最多15步，temperature=1.0、top-p=0.9。它没有训练新的3B摘要器，也没有实现历史区域RAG。
原始截图为1920×1080。可访问性树先按可见性、标签与属性过滤成表格，提供元素位置。若用户前端使用YOLO/OCR，主实验不能与“有内部坐标的树输入”混为完全相同条件。
4. 指标与范围
原文称Success Rate，由任务末执行评估得分汇总；部分函数可给0到1的部分分数。应报告具体评分口径，不把所有分数称为严格二值全完成率。模型的DONE只是终止动作，不能替代评估器。15步结果也不能直接与后来50/100步的排行榜比较。
5. 结果、历史分析与附录
表5的原论文代表值：
| 观察与模型 | 全部任务得分（%） |
|---|---|
| 可访问性树，GPT-4 | 12.24 |
| 纯截图，GPT-4V | 5.26 |
| 截图＋树，GPT-4V | 12.17 |
| SoM，GPT-4V | 11.77 |
| 人类 | 72.36 |
这些是2024论文配置，不是2026模型水平。原文§5.2在10%子集上改变历史长度与分辨率：更多文字/SoM历史有帮助，纯截图历史未呈现同样结论；因此不能概括成“越多旧图越好”。它同时指出长元素表的上下文成本。
附录A给观察/动作接口，B给初态与检查，C给模型提示、树过滤、SoM和完整结果，D给重复动作、窗口干扰、定位与应用知识错误。环境版本、文件格式、网页变化及模型停止方式均会影响成绩。
6. 对本课题的用途
适合桌面在线迁移评测，与原目录Y26-153的OSWorld2.0区分。先选确实依赖历史的任务子集，再同时报告全体结果；固定输入权限、执行模型、步数和终态检查，避免将更大预算的收益当记忆创新。历史原图中的坐标只能帮助识别证据，实际动作必须绑定当前截图。
7. 来源与核查范围
原文完整表格附件保留本版本可抽取的正文及附录表；正文解释关键比较，不把异构数据集或不同模型拼为统一排行榜。
原论文：章节、图注与来源 / Original paper。章节包括：1 Introduction；2 OSWorld Environment；2.1 Task Definition；2.2 Real Computer Environment Infrastructure；2.2.1 Overview；2.2.2 Initial Task Environment Setup；2.2.3 Execution-Based Evaluation；2.3 Observation Space；2.4 Action Space；3 OSWorld Benchmark；3.1 Operating System and Software Environments；3.2 Tasks；3.3 Data Statistics；3.4 Human Performance；4 Benchmarking LLM and VLM Agent Baselines；4.1 LLM and VLM Agent Baselines；4.2 Results；5 Analysis；5.1 Performance by Task Difficulty, Feasibility and App Involved；5.2 Performance by Multimodal Observation Variances；5.3 Performance across Different Operating Systems；5.4 Qualitative Analysis；6 Related Work；7 Conclusion and Future Work；Acknowledgements。
Open this note in the interactive notebook (comments, hooks) → · All notes