You Only Look at Screens: Multimodal Chain-of-Action Agents

A smartphone command usually requires several interactions, but the next action must be chosen from the current screen. Systems that translate each interface into text through separate parsers can lose visual information and introduce additional processing dependencies. Auto-GUI investigates whether a compact multimodal model can learn this decision directly while representing the task's temporal structure.

A screenshot-based encoder-decoder agent that combines visual observations, a user goal, and structured previous actions without a runtime UI parser.

A chain-of-action representation with action history on the input side and an anticipated sequence of action types on the output side.

An open implementation with action-coordinate normalization and experiments isolating visual features, model capacity, and history.

At each step, a frozen image encoder extracts screenshot features. A language encoder processes the instruction and previous action records, including action type, touch point, lift point, and typed text. A trainable projection aligns visual features with the language representation; attention and a learned gate combine the two channels before decoding.

The decoder produces an action plan followed by an action decision. The plan lists anticipated action types rather than executable coordinates for every future screen. The decision supplies the current operation and its parameters. Thus, the plan is auxiliary structure for selecting the next action; the environment still has to provide a new observation before subsequent decisions.

Touch and lift coordinates express both taps and swipes. Coordinates are normalized across screen sizes, and preprocessing standardizes common gesture patterns to reduce variation in demonstrations. The released workflow prepares screenshot features and trajectory records before model training, then decodes action strings for evaluation. This is the Auto-GUI system from 'You Only Look at Screens: Multimodal Chain-of-Action Agents,' not the similarly named later AutoGUI dataset.

Android in the Wild supplies recorded tasks covering general phone use, app installation, Google apps, single-step interactions, and web shopping. Episodes are divided into training, validation, and test sets. The principal measure is screen-wise action matching, with rules for action types, gesture locations or target regions, and swipe direction.

Comparisons include prompted language models, a fine-tuned language model, and behavioral-cloning agents. Ablations change the image features, language-model scale, and available action history to test what each component contributes. The protocol evaluates predictions at demonstration states rather than rolling out complete tasks in a live Android environment.