SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

A model may understand a user request yet fail to identify the corresponding control in a screenshot. GUI grounding differs from recognizing objects in natural photographs: targets can be small text labels, unfamiliar icons, or visually similar widgets. SeeClick treats this localization problem as a capability that requires dedicated supervision and a benchmark covering multiple interface platforms.

A GUI grounding pretraining recipe that maps screenshot-and-description pairs to coordinates through ordinary language generation.

Automatically collected web examples and reorganized mobile interface data for grounding, text recognition, widget descriptions, and screen summaries.

ScreenSpot, a human-annotated localization benchmark covering mobile, desktop, and web interfaces with textual and non-textual targets.

Released model, data preparation, training, and evaluation resources connecting grounding pretraining to visual GUI agent tasks.

A grounding example supplies the screenshot and a description of the desired element. The model outputs a point or bounding box as numerical text. This keeps localization within the language-model generation interface rather than introducing a separate vocabulary of coordinate tokens or requiring an external element detector at inference.

Web supervision comes from rendered pages whose visible text and descriptive HTML attributes can be associated with element locations. The training mixture also reverses widget-captioning examples to create description-to-location tasks and incorporates mobile UI grounding and summarization resources. General vision-language instruction examples remain in the mixture alongside the GUI-specific material.

The released implementation adapts Qwen-VL with GUI grounding data and then fine-tunes for particular agent benchmarks. In the agent setting, the prompt adds the task and previous actions to the screenshot, and the model predicts the next operation with its target. The crawler and mobile preprocessing produce training labels; they are not a requirement to supply DOM trees or accessibility metadata to the deployed visual agent.

ScreenSpot tests whether the predicted click lies inside the annotated target box, with breakdowns by platform and by text versus icons or widgets. Its role is to measure localization given a target instruction, separately from deciding which action advances an overall task.

Downstream experiments use MiniWob, Android in the Wild, and Mind2Web. They evaluate task completion in the interactive MiniWob setting and benchmark action or step correctness on recorded mobile and web demonstrations. Comparisons with the base vision-language model and GUI-trained alternatives examine whether grounding training transfers to action prediction.