Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

GUI agents trained in a narrow benchmark environment may lack the execution access, device reliability, or experience needed for real workflows. The report addresses mobile, desktop, browser, and information-seeking tasks together, including workflows that cross platforms. Its scope includes the surrounding runtime and training infrastructure as well as the policy model.

A unified environment interface supports sandbox systems and real mobile devices while exposing graphical operations, command execution, API calls, and requests for user input.

An agent-driven data pipeline connects task synthesis, environment preparation, trajectory collection, verification, failure diagnosis, and subsequent training iterations.

The training recipe combines domain-specific supervised learning, action-level correction, and online reinforcement learning; MobileWorld-Real supplies a real-device evaluation setting.

At each decision, the model receives the instruction, interaction history, and available visual or structured observations. It can emit one operation or an ordered batch. Environment adapters translate these operations into native actions and return screenshots, command results, or service responses through a shared lifecycle.

The real-device runtime tracks device, application, account, network, and display availability. It routes work to usable targets and supports handover for information or operations requiring the user. Trajectory review distinguishes policy failures from environment failures so infrastructure problems can be repaired separately.

The data pipeline synthesizes tasks and their initial states. For checkable outcomes, it constructs executable verifiers. A visual judge also identifies useful segments in successful or failed trajectories for supervised learning. Failure diagnoses guide the next round of task generation.

Domain experts are supervised separately and merged into a unified checkpoint. Action-level reinforcement learning targets recurring local errors with operation-specific rewards. Online reinforcement learning then uses complete environment interactions and task outcomes to train longer decision sequences. A harness retains workflow state across services and platforms.

MobileWorld, MobileWorld-Real, and AndroidDaily cover simulated and real-device mobile tasks. OSWorld-Verified and OSWorld-v2 test desktop execution; WebArena tests browser tasks; BrowseComp variants test information seeking. Grounding and general-capability suites are evaluated separately.

Protocols use benchmark-specific completion or partial-progress measures. MobileWorld-Real records environment errors separately and excludes them from its success denominator. The report also analyzes GUI–CLI coordination, batched execution, and training-stage behavior. Proactive-service demonstrations are distinguished from these benchmark protocols.