AutoDroid: LLM-powered Task Automation in Android

General language knowledge does not necessarily reveal how a particular Android application exposes a function. A task may require navigating through controls whose labels do not resemble the user's goal. Meanwhile, raw UI hierarchies are lengthy and repeated model calls are costly. AutoDroid combines application exploration, compact UI descriptions, and selective memory retrieval to address these operational gaps.

A task-oriented interface representation that converts Android UI structure into compact, actionable HTML-like elements.

An exploration-based memory pipeline that records UI transitions and synthesizes descriptions of functions reachable through those transitions.

A memory-augmented execution loop with techniques for reducing repeated UI processing and language-model query cost.

A system and Android task benchmark used to compare action decisions, complete task execution, and the cost of different configurations.

Before user-task execution, an explorer interacts with the app and builds a UI Transition Graph. Screens are nodes and recorded actions connect them. A language model summarizes functions associated with UI elements and screens, producing simulated tasks together with the paths that reach them. This memory records operational knowledge about the analyzed application rather than relying on general model pretraining alone.

During execution, the user's goal is matched to simulated tasks through sentence embeddings. Relevant paths and UI functions become hints in the current prompt. The prompt also includes the actual screen representation and prior actions. The model selects an element identifier, an operation, and any input text from a constrained output format; an executor translates that selection into an Android interaction.

The interface representation prunes irrelevant structure, groups functionally equivalent elements, and can inspect scrollable content before requesting a decision. The system also includes prompt filtering and confirmation handling for selected sensitive operations. Its released workflow separates app-memory preparation from online task execution, making exploration effort and memory reuse part of the deployment assumptions rather than hidden capabilities of a screenshot-only model.

DroidTask provides Android task instructions, reference action traces, and application environments for exploration. The paper evaluates different language-model backbones using action accuracy and task completion. Comparisons and ablations investigate app memory, UI representation, exploration coverage, and optimizations that change the number or size of model queries.

Cost analyses consider model interaction and the work required to prepare application knowledge. The experiments assume that the apps can be explored and their interface information analyzed. They test the configured system on its benchmark, rather than showing that a model can infer the same app-specific transitions from an arbitrary screenshot without prior analysis.