Android in the Zoo: Chain-of-Action-Thought for GUI Agents

A screenshot and a coordinate-labelled action omit much of the context that connects a user request to an interaction. The current state may depend on an earlier action, and a raw action history does not explain what changed. The paper proposes semantic annotations that make these relationships explicit for GUI learning and prompting.

Chain-of-Action-Thought defines separate components for screen description, action reasoning, next-action description, and the result of an executed action.

The Android-in-the-Zoo dataset samples and validates Android-in-the-Wild demonstrations, then adds model-generated semantic annotations with expert correction.

Evaluation procedures examine both individual action matching and progress through a reference episode, comparing prompting and fine-tuning uses of the representation.

Dataset construction first samples instructions and episodes using strategies tailored to the source subsets. These strategies reduce repeated templates and balance categories. Human annotators then check whether the recorded episode actually fulfils its instruction, so semantic annotation is built on verified demonstrations.

A multimodal model generates the semantic components with different input information for each task. Screen descriptions summarize visible content. Action reasoning connects the request, current screen, and history. Next-action descriptions use the known demonstration action to state the intended operation in readable form.

Action Result is retrospective: it summarizes the change between screenshots before and after an executed action. At the next step, the previous result can join earlier action descriptions as history. It is therefore not simply a prediction of a future outcome. Experts correct inconsistent action descriptions and trigger regeneration of dependent annotations.

The components can be combined in prompts for larger multimodal models or used to supervise a smaller model. Their explicit fields allow experiments to vary the semantic context supplied for action prediction without changing the underlying recorded interaction.

Zero-shot experiments compare the proposed context representation with standard prompting, action-history prompting, and chain-of-thought prompting. Fine-tuning experiments test learning from the annotated mobile demonstrations. Component comparisons examine the role of different semantic fields.

Screen-wise action matching checks the action type and relevant arguments, such as position, scroll direction, text, or button. The episodic goal-progress metric records where the first error occurs relative to the reference sequence. These are demonstration-based measurements rather than unrestricted live-app completion tests.