Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

A GUI planner can describe the intended operation without locating the correct control. Grounding must connect that description to a screen position even when the description refers to appearance, relative location, or function. Screenshots also vary in size and orientation, while HTML and accessibility metadata are not consistently available across platforms.

A hybrid data-construction pipeline pairs rendered webpages and element coordinates with visual, positional, and functional referring expressions, including descriptions that combine these cues.

A grounding model adapts LLaVA-NeXT to high-resolution screenshots and directly generates pixel coordinates for the described target.

The SeeAct-V framework separates screenshot-based planning from coordinate grounding, allowing different planners to use the same grounding component.

The main synthesis pipeline matches HTML elements to their rendered bounding boxes. Rules extract attributes and construct spatial relationships; a multimodal model supplements incomplete metadata with visual or functional descriptions. A language model makes generated descriptions more concise. The target label is the element's center point.

The final training mixture combines these web examples with directly generated web descriptions and existing Android grounding datasets. Web synthesis supplies most of the examples, but the trained model is not based exclusively on web data.

UGround processes a screenshot with an instruction describing the target and autoregressively returns unnormalized pixel coordinates. Its image pipeline expands the supported tiling configurations for large portrait and landscape screenshots. Within SeeAct-V, a planner produces the next operation and target description, UGround locates that target, and the environment executes the resulting action before providing another screenshot.

ScreenSpot tests element localization across mobile, desktop, and web interfaces. Offline evaluations on Multimodal-Mind2Web, AndroidControl, and OmniAct assess web and mobile next actions and desktop automation scripts against annotated references. Online evaluations use Mind2Web-Live and AndroidWorld to test task execution in changing environments.

Data-scaling and source-mixture ablations examine what the training corpus contributes. Further analysis studies referring-expression types and grounding errors. These protocols distinguish isolated target localization from the additional planning and interaction demands of a complete agent.