TinyClick: Single-Turn Agent for Empowering GUI Automation

GUI target localization can be computationally expensive when assigned to a large multimodal model. TinyClick studies a narrower task: locating the interface element specified by one instruction. It asks how a compact pretrained vision model can learn that mapping using related visual tasks and enriched annotations.

A compact Florence-2-Base model is adapted for instruction-conditioned GUI point prediction using its existing visual and coordinate-token representations.

A multitask training mixture combines command following with element captioning, localization, clickable-object detection, and screen question answering.

Additional model-generated descriptions supply element purpose and expected effects, complementing existing bounding boxes, commands, and interface metadata.

Each action-training example pairs a screenshot and a command with the location of the intended control. Florence-2 encodes the image and generates the spatial answer through its language decoder. Coordinates use the model's dedicated location-token representation, allowing the same architecture to handle both descriptions and spatial outputs.

Auxiliary tasks connect different views of the same interface element. Given its location, the model may describe the element or its purpose; given a description, it must find the element. Detection examples identify clickable controls, and question-answering examples require interpretation of the screen's content.

The authors combine public GUI datasets and supplement their annotations with a larger multimodal model. Software metadata provides detection labels where available. The training recipe removes evaluation examples from contributing data sources and varies the data mixture. This supervision develops visual and functional recognition; it does not train TinyClick to independently plan an entire sequence of application operations.

ScreenSpot tests point prediction across mobile, desktop, and web screenshots, including text and icon targets. The OmniAct evaluation uses individual annotated boxes and descriptions, rather than the dataset's full multi-turn task commands. Correctness depends on the predicted point falling within the intended target.

Ablations compare multitask training with command-only training and examine alternative data mixtures. Error analysis distinguishes incorrect element identification from imprecise pointing. The protocol therefore measures single-step grounding, without attributing end-to-end navigation or recovery capabilities to the model.