GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices

Mobile workflows often require carrying information from one application into another. The current screenshot may omit details needed for the next action, while retaining every earlier screenshot is costly. GUIOdyssey addresses both the shortage of cross-app demonstrations and the representation of visual history in an agent.

A mobile dataset records human demonstrations across applications and devices, with high-level instructions and additional semantic annotations for individual steps.

OdysseyAgent adds a learnable history resampler to a vision-language model, compressing past screenshots before predicting the next action.

Separate random, app, task, and device splits distinguish in-domain evaluation from several forms of generalization.

Human demonstrators carry out designed cross-app tasks on mobile devices. The recordings preserve screenshots, actions, and the task instruction. Additional annotations describe finer-grained operations and interaction semantics, supporting experiments with either high-level goals or more explicit step-level instructions.

OdysseyAgent builds on Qwen-VL-Chat, retaining its vision encoder, vision-language adapter, and language model. A history resampler uses learnable query embeddings to attend to visual tokens from earlier screenshots. This cross-attention converts a potentially long visual history into a shorter set of representations.

The compressed history is combined with the current screenshot, user instruction, and preceding actions. The language model predicts the next operation as a token sequence. Training adapts the vision-language adapter, history resampler, and language model to the demonstrations. The method therefore combines visual memory with the textual record of earlier actions, rather than treating them as interchangeable sources of context.

The ICCV version evaluates held-out recordings under random, unseen-app, unseen-task, and unseen-device conditions. It compares high-level and low-level instruction settings and tests the addition of semantic annotations.

Action Matching Score checks operation type and operation-specific arguments. Pointing actions use spatial or target-region criteria, scrolling checks direction, and typing uses string similarity. Episode success requires the entire annotated action sequence to match, making it an offline sequence metric rather than an independently checked final application state.

Ablations remove history components, vary the amount of history, and compare the resampler with direct multi-image input. These tests investigate which context helps action prediction and the computational cost of representing it.