Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots

Overall completion scores do not explain whether a mobile agent failed because it misread the goal, could not access the UI, chose the wrong control, or could not execute its decision. This study develops everyday Android tasks and a controlled modality comparison to connect measured behavior with concrete interface and agent failures. It also examines the resource implications of adding visual observations.

DailyDroid, a task suite spanning everyday application categories and difficulty levels defined by interaction complexity.

A shared experimental framework comparing screentext with screentext plus screenshots under the same task and execution setup.

A failure handbook separating system-level interface problems from agent-level interpretation and decision problems.

A protocol combining execution logs, resource measurements, and manual outcome verification rather than relying only on agent-reported success.

The study adapts AutoTask, whose Android-side recording service and computer-side backend support repeated observation and execution. Accessibility information is converted into simplified HTML-like elements with identifiers. The backend receives the task and current representation, requests an action, executes it through the device interface, and continues from the resulting state. Backtracking and reflection are available within this inherited control framework.

For the modality experiment, the authors disable contextual knowledge injection and memory, and add screenshot capture to AutoTask. One condition sends only the structured screentext; the other sends that same screentext together with the screenshot. Both conditions retain the shared parser and action-grounding mechanism. The comparison therefore concerns the added screenshot channel, not a contest between a pure visual agent and a text-only agent.

Tasks are authored and pilot-tested across application categories, with difficulty based on navigation depth, decision points, and state dependencies. After execution, the authors inspect prompts, responses, screen records, and action traces. They manually verify outcomes and inductively organize recurring failures into diagnostic categories.

GPT-4o and o4-mini are tested under both input conditions with a common emulator setup and a bounded action budget. Measurements cover task success, action count, total runtime, model waiting time, device-side time, token usage, and estimated API cost.

Failure coding examines UI retrieval, parsing, interface logic, execution, and agent reasoning or interpretation. The framework did not support a screenshot-only condition, and the study does not repeat every task-condition pair across multiple runs. These protocol details define which modality and reliability questions the experiment measures.