OmniParser for Pure Vision Based GUI Agent

A multimodal planner can describe a sensible next step while pointing to the wrong location. Supplying boxes from a browser DOM or mobile UI hierarchy can simplify grounding, but those structures are not consistently available across interfaces. OmniParser addresses the perception stage by deriving candidate elements and their descriptions from the screenshot itself.

A screenshot parsing pipeline combining interactable-region detection, icon description, and optical character recognition.

Web-derived supervision for training an interactable-region detector using element boxes available during data collection.

A structured output containing element identifiers, locations, and semantic information, paired with an annotated screenshot for downstream reasoning.

An open parser implementation and experiments integrating the original system with a screenshot-aware action planner.

The input screenshot is processed by an interactable-icon detector and an OCR component. Their boxes are merged, with overlapping detections reconciled so the same visible element does not create redundant targets. Each retained region receives an identifier that links its textual representation to its location in the image.

A captioning component supplies functional descriptions for icons whose meaning may not be evident from OCR text. The parser then produces a DOM-like list of regions and a screenshot annotated with corresponding marks. This representation gives the downstream model explicit targets and semantic hints instead of asking it to invent precise coordinates without intermediate candidates.

In the original agent integration, GPT-4V receives the parsed information and screenshot context, selects an operation and target identifier, and the system maps the identifier back to the target region. The released parser exposes image-processing components and utilities independently of that planner. DOM-derived labels are used to create detector training data; runtime parsing operates on pixels. Later repository releases add components and models, which should be distinguished from the original paper's evaluation.

ScreenSpot tests whether the agent can ground an instruction on the appropriate screen element. Mind2Web and Android in the Wild evaluate next-action prediction on recorded web and mobile tasks. The study compares the original parser-assisted agent with GPT-4V configurations using other representations or grounding arrangements.

The evaluation asks whether better detected regions and element descriptions help the downstream planner choose executable actions. It concerns the combined parser-and-planner setup, not parsed text as a complete replacement for visual context. The benchmark protocols distinguish localization from action correctness and do not constitute a general live-task completion evaluation.