OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

Training data for GUI agents is unevenly distributed across platforms, with much more accessible web data than desktop data. Existing trajectories also encode equivalent operations differently, such as a desktop click and a mobile tap. OS-Atlas addresses both problems: collecting diverse interface supervision and making actions compatible enough for joint training and transfer to unfamiliar environments.

An automated grounding-data collection infrastructure covering web, Android, Windows, macOS, and Linux interfaces.

A grounding-then-action training pipeline that separates element localization from instruction-conditioned action prediction.

A unified action space with shared basic operations and explicitly described custom actions for platform-specific behavior.

Released models and data, plus corrected ScreenSpot annotations and experiments separating grounding, action prediction, and downstream adaptation.

The collection pipeline renders web pages and links visible elements to their descriptions and coordinates. Desktop and mobile collectors explore interfaces using platform-specific accessibility tools. Screenshots and labels are filtered before training. Additional instruction-grounding examples describe the purpose of an action using its task context and before-and-after screenshots.

Grounding pretraining asks a vision-language model to locate an element from a screenshot and referring expression or instruction, producing OS-Atlas-Base. Action fine-tuning then uses task instructions, screenshots, and available history to predict thoughts, action types, and arguments. The released model family offers grounding and action variants rather than treating these two training stages as interchangeable outputs.

The action vocabulary maps common operations to shared click, type, and scroll forms. Custom actions, such as opening an application or dragging an object, are introduced through descriptions appropriate to the environment. This representation lets training examples from different interfaces share an output format while preserving operations that are not available everywhere. A grounding model can also be inserted beneath a separate planner to supply coordinates for its chosen action.

ScreenSpot and its corrected version test point localization and bounding-box overlap. OSWorld evaluates the grounding model within an interactive agent using an external planner. These are different uses of the released models: locating a requested target versus completing a sequence of desktop actions.

Agent experiments cover AndroidControl, GUI-Odyssey, GUI-Act-Web, and the web and desktop portions of OmniAct. They measure action-type correctness, grounding, and step success under out-of-distribution and downstream fine-tuning settings. Ablations examine data scale, platform coverage, instruction-grounding examples, and action-space unification.