Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

GUI datasets differ in their observations, action names, and assumptions about planning. A model trained only to localize a supplied target does not automatically know which target should come next in a longer task. Aguvis develops a common visual interaction framework and training mixture intended to connect these grounding and decision-making capabilities within an open model.

A screenshot-based interaction framework using a common action format with extensions for platform-specific operations.

Aguvis Data Collection, combining converted action datasets, synthetic grounding supervision, and trajectory annotations with explicit intermediate reasoning.

A two-stage training recipe that first learns grounding and then adds planning and reasoning over GUI tasks.

Released checkpoints, data, and training resources for using Aguvis either as a grounding component or as an agent that produces its own plans.

At an interaction step, the model receives the user goal, the current screenshot, and relevant history. It can generate an inner monologue describing the current situation and intended next action, followed by an executable command. The action format is based on PyAutoGUI, with plugins for mobile gestures, platform controls, completion signals, and other operations that need an explicit extension.

Data construction first translates existing GUI trajectories into this common format. Interface metadata also supports template-generated instruction-and-action pairs. A separate annotation process enriches trajectories with intermediate reasoning and action instructions. Closed-source vision-language models assist this annotation stage; the resulting released model can generate its own thoughts and actions at inference.

Grounding training develops the mapping from an instruction and screenshot to an interface action. The subsequent stage mixes reasoning-rich trajectories with simpler action instructions, teaching the model different levels of planning detail. The project exposes both grounding-oriented and full agent checkpoints, allowing experiments to separate a planner's choice of target from the model's ability to execute that choice visually.

ScreenSpot tests direct grounding and a setting that includes planning before localization. Multimodal-Mind2Web and AndroidControl test recorded action prediction, including high-level task instructions and lower-level action instructions where applicable. These measures concern the next action at a supplied state.

Mind2Web-Live, AndroidWorld, and MobileMiniWoB test interactive task execution. The evaluation distinguishes configurations using an external planner from those using Aguvis for planning and grounding together. Ablations investigate training stages, inner-monologue annotations, cross-platform data, and observation-processing efficiency; they do not turn benchmark completion into a general reliability guarantee.