ShowUI: One Vision-Language-Action Model for GUI Visual Agent

GUI agents must process screenshots with small actionable elements alongside large areas of repetitive background. Long visual histories further increase the sequence length. At the same time, actions differ across devices and depend on what happened earlier. ShowUI addresses the representation of both screen content and interaction history.

A UI-connected graph groups visually similar neighboring patches and guides token selection while retaining each selected patch's original positional information.

Interleaved vision-language-action streaming represents navigation histories and supports multiple grounding queries associated with a shared screenshot.

A curated instruction-tuning mixture balances grounding and navigation data across web, desktop, and mobile interfaces, including varied descriptions of target elements.

The screenshot is divided into patches. A graph links neighboring patches when their RGB content is sufficiently similar, and a union-find procedure identifies connected components. Patches in the same component are treated as visually redundant. During training, the model skips a portion of their tokens while leaving single-patch components intact.

Selected tokens keep their original position embeddings. This differs from replacing an entire component with a pooled token, which changes the spatial representation. The selection procedure adds no trainable routing network and can also be applied during inference.

Actions use a common structured format specifying the operation, a value when needed, and a position. The system prompt describes the available action space. Navigation examples alternate screenshots and preceding actions so the next prediction can use both visual state and interaction history; grounding examples reuse a screenshot across several query–answer pairs.

Instruction tuning builds on Qwen2-VL. Data curation emphasizes interactive visual elements, augments desktop target descriptions with appearance, spatial, and intention-based queries, and balances sources whose sizes differ substantially.

ScreenSpot evaluates target grounding across platforms. Navigation experiments use Mind2Web, AITW, and MiniWoB, applying each benchmark's action-matching or task-completion protocol. The study also measures the amount of visual processing required by alternative representations.

Ablations vary token-selection ratios and compare selection with merging or other reductions. They examine data choices and the contribution of observation–action history, connecting efficiency measurements to the grounding and navigation tasks being performed.