OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

A computer assistant may need to move among applications, manipulate files, and continue work from a partially prepared state. Evaluating only a predicted action against a recorded click cannot establish whether that work was completed. OSWorld addresses the infrastructure and task-specific validation needed to test such interactions on functioning computers.

A computer environment connects an agent to virtual machines running real applications, with support for different observation and action configurations.

Tasks include explicit initial-state preparation and custom evaluators that inspect the files, application settings, or other state relevant to the user's request.

The benchmark spans individual applications, file operations, and multi-application workflows, with human evaluation and diagnostic studies of agent observations and errors.

A task configuration specifies the instruction, required files and application state, and the checks that define completion. The task manager initializes that state before interaction begins. This allows an episode to start from realistic work in progress rather than only from an empty desktop.

The agent observes the machine through screenshots, accessibility information, or a selected combination. A controller executes mouse and keyboard operations inside the virtual machine and returns the resulting observation. The infrastructure supports isolated environments running in parallel.

After the episode, the evaluator can post-process application state, retrieve relevant files or information, and apply a task-specific comparison. For example, a document-editing task requires checking the edited artifact rather than accepting the agent's declaration that it has finished. Some checks use application configuration or external information, so the validation logic varies with the task.

The environment, tasks, and baseline agents are separate components. This permits a new agent to use the same task setup and validator, or a new task to supply its own setup and evaluation functions while reusing the execution machinery.

The original paper evaluates computer tasks involving desktop software, browser applications, file handling, and cross-application workflows. Completion is determined by the supplied execution-based evaluator; human runs provide an additional reference for task feasibility and difficulty.

Experiments vary screenshot and accessibility inputs, resolution, history length, and interface layout. They examine application categories, workflow complexity, and error types. This note describes the original paper's protocol; later OSWorld-Verified and OSWorld-v2 releases have distinct evaluation configurations.