CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

Smartphone agents must resolve small interface details and produce commands whose parameters agree with the intended operation. A coarse image representation may miss an icon or label, while an undifferentiated action format asks the model to generate fields that are irrelevant or inconsistent. CoCo-Agent addresses these perception and output-representation problems within a multimodal action model.

Comprehensive Environment Perception, which combines the current screenshot, fine-grained layout descriptions, the goal, and action history.

Conditional Action Prediction, which separates deciding an action type from generating the target and parameters required by that type.

Action normalization that connects detected interface elements with click targets and expresses scrolling through direction-specific operations.

An open LLaVA-based implementation evaluated on both smartphone action demonstrations and a dialogue-enabled GUI benchmark.

The visual-language backbone uses a vision encoder and projection layer to bring screenshot features into a language model. The text input adds the user goal, previous actions, and a list of interface items with locations. OCR and icon-recognition tools provide this detailed layout channel, supplementing information that the global screenshot representation may not resolve.

The model first predicts what kind of operation is required. Its continuation supplies only the relevant target information: a click can name an item and its location, a scroll specifies a direction, and text entry supplies a string. The representation separates clicks associated with detected elements from taps that are not associated with a detected box.

Training normalizes element clicks to box centers and scroll gestures to standard directions, reducing inconsistent surface forms in demonstrations. Generated descriptions are parsed back into the benchmark action format for execution or scoring. The released resources include model and run examples built around this multimodal representation; the method deliberately combines pixels with tool-derived layout text rather than treating either channel as the whole observation.

Android in the Wild tests next-action prediction across its application subsets, including separate-subset and unified training configurations. Action correctness requires the relevant fields to agree with the annotation, using target-box or coordinate-tolerance rules for taps and directional matching for scrolls.

META-GUI adds dialogue-enabled interactions. Its evaluation includes text-entry F1 and response-generation BLEU alongside action fields. Component ablations examine detailed layout information, historical actions, and conditional action modeling. These tests evaluate predictions from recorded interaction states and dialogue context; they are not measurements of unrestricted online recovery from the agent's own errors.