AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

Recorded action datasets cannot show how an agent responds when its own action changes the screen, and live-app evaluations can be difficult to repeat. AndroidWorld addresses the need for an executable mobile environment whose tasks can vary while preserving valid initialization and success criteria. The benchmark concerns complete interactions with applications, rather than only matching isolated demonstration actions.

An Android environment exposing screenshots, accessibility information, and an action interface for agents.

Parameterized task definitions with setup, reward evaluation, and teardown tied to controlled application state.

Reusable validators that inspect files, databases, settings, or UI elements and support tasks composed from simpler objectives.

A released benchmark implementation and baseline agents, including M3A, for studying observation formats and prompting designs.

The environment runs a fixed Android emulator configuration with controlled application versions. A task samples parameters under a specified random seed and prepares the corresponding initial state. The resulting instruction may refer to particular names, file contents, appointments, or other task-specific values, so different instances require more than replaying a fixed coordinate sequence.

An agent obtains the current screenshot and, when its configuration uses it, an accessibility-tree representation. It acts through gestures, text entry, navigation controls, and the supported function interface. The environment applies the action and returns a changed observation. Completion is checked against the requested state, using application databases or files where possible and UI validation where necessary.

The M3A baseline selects an action from the available operations and then inspects the transition. Its reflection stage summarizes what happened and provides a history for the next decision. The benchmark also supports simpler prompting and an adapted SeeAct baseline. In the released task API, initialization and validation belong to task code, keeping the evaluator separate from an agent's own claim that it has finished.

Experiments measure task success on AndroidWorld and MobileMiniWoB under defined interaction budgets. Agent comparisons vary the underlying language model, text-only versus screenshot-assisted observations, operating guidelines, and reflection. Human task attempts provide a reference for task difficulty and the interaction budgets used in the study.

Further analyses inspect failures and sensitivity to task parameters. Task seeds, application versions, and reset procedures matter because the goal is to compare agents on controlled yet varied instances. The state-based validators assess specified outcomes; their use does not imply that every behavior outside those checks has been evaluated.