GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

Generating coordinates as ordinary text leaves the relationship between numeric tokens and visual features implicit. It also assigns a single target point to controls that admit many valid clicks. GUI-Actor studies a spatial prediction interface that works directly with the patch representation of a vision-language model.

An action head connects a dedicated actor token to screenshot patches, making the grounding prediction an attention map over visual regions.

Multi-patch supervision treats patches overlapping the target element as valid, representing the spatial extent of an actionable control.

A separately trained grounding verifier checks candidate locations, while a frozen-backbone training setting isolates the capability of the new action head.

The language model processes the screenshot and instruction, then generates an action containing special actor tokens in place of the coordinate string. The final hidden state of the actor token serves as a contextual query for grounding.

The action head first contextualizes visual patches with self-attention. Separate projection networks map the actor representation and patch features into a shared space. Their similarities form a probability distribution over the screenshot. Training combines ordinary next-token prediction with an attention objective whose target distributes probability across patches covered by the annotated bounding box.

At inference, high-attention patches provide candidate locations. The verifier receives the instruction and a screenshot marked at each candidate and predicts whether the marked point satisfies the instruction. It is trained using correct targets and incorrect locations from the same interface. Candidate checking follows attention order and can stop when a location passes the confidence criterion.

ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro test whether the final click lies inside the annotated target. The professional-software setting probes unfamiliar layouts and high-resolution screens. Online OSWorld-W experiments examine grounding within an agent execution loop.

Controlled comparisons vary supervision, training-data scale, and whether the backbone is frozen. Verifier ablations test candidate selection and aggregation, including application to other grounding models. The measurements separate architectural choices in localization from broader model training.