UI-TARS: Pioneering Automated GUI Interaction with Native Agents

A GUI agent must read dense screens, ground actions, maintain progress over time, and respond to states produced by its own mistakes. Modular systems often assign these functions to separate components, while action-only demonstrations provide limited examples of recovery. The original UI-TARS report investigates learning these behaviors within one vision-language model through staged data construction and training.

A native agent formulation that conditions thought-and-action generation on the task, recent screenshots, and earlier interaction history.

Perception and action datasets covering element descriptions, dense screen captions, state changes, visual question answering, and shared GUI operations.

Thought annotation and bootstrapping procedures that enrich demonstrations with intermediate reasoning and select usable thought-action pairs.

Iterative online trace collection and reflection training that include corrected decisions and continuations from states reached after errors.

The model reads screenshots directly and generates a thought followed by an action. Recent visual observations are retained alongside the textual history of thoughts and actions. Platform-specific prompts specify the available desktop or mobile commands. The released parser converts generated operations and normalized coordinates into executable calls for the original screen dimensions.

Perception training links interface images to element properties, spatial descriptions, and screen-level explanations. Action training normalizes equivalent operations across platforms and learns sequences rather than unrelated clicks. Thought augmentation first annotates demonstrations with reasoning; bootstrapping additionally samples thought-action pairs without revealing the reference action, retaining pairs whose action matches the training target.

For further training, agents execute task instructions in interactive environments and produce new trajectories. Filtering and reflection identify usable traces, corrected actions, and recovery continuations. Supervised learning and preference optimization incorporate these materials into subsequent checkpoints. This is a training process: the description does not assume that model weights update during every deployed user session.

The report evaluates interface perception, visual grounding, offline action prediction, and online task completion separately. Representative suites include VisualWebBench and ScreenQA for perception, ScreenSpot variants for localization, AndroidControl and GUI-Odyssey for recorded mobile actions, and OSWorld and AndroidWorld for interaction with changing environments.

The study varies model scale, training components, reasoning use, and sampling budgets to examine which capabilities each contributes. Grounding checks target-region correctness; offline tests require the action and arguments to match annotations; online tests use environment task evaluators. This note concerns the original report and does not merge experiments from later numbered UI-TARS releases.