GPT-4V(ision) is a Generalist Web Agent, if Grounded

Describing a browser action is different from identifying the exact page element on which it should execute. A screenshot-aware model may produce a plausible instruction without reliably mapping it to a DOM element. SeeAct isolates action generation from grounding so their errors can be studied separately, while also examining the difference between cached demonstrations and live website interaction.

A web-agent pipeline that generates an intended action from screenshot context before converting it into an executable browser operation.

A comparison of element-attribute matching, textual candidate selection, and visual mark selection as grounding strategies.

A human-assisted oracle-grounding condition for estimating the contribution of grounding errors without replacing the model's intended action.

Offline and online evaluation resources, with a released browser implementation that exposes action prediction and execution as separate stages.

The action-generation prompt supplies the task, current screenshot, and previous actions. GPT-4V analyzes this context and describes the next operation in language. The intended action must then be resolved into a target element, operation type, and any value required by the operation, such as text to enter.

Grounding via attributes searches for an element matching the generated type and text. Grounding via textual choices presents ranked HTML candidates as a multiple-choice question. Grounding via image annotation overlays candidate boxes and labels on the screenshot and asks for the target label. These alternatives change how the same intended action is connected to an actual page element.

The oracle condition asks a human to identify the action the model meant, provided its description contains the needed information. It does not authorize the human to invent a better task plan. In the executable pipeline, the selected element and operation are passed to browser automation, a fresh observation is collected, and the process repeats. The released interface separates prediction from execution so grounding or planning components can be substituted explicitly.

Offline experiments use Mind2Web task splits to measure element selection, operation correctness, and step success on cached states. Live-website experiments instead execute action sequences and assess whether the task is completed, allowing valid alternative routes that may differ from a recorded demonstration.

The study compares grounding strategies, oracle assistance, and model configurations, and analyzes errors involving element descriptions and visual labels. Its oracle setting is an intervention on the grounding stage, not a fully automatic agent result. The online-versus-offline comparison tests the consequences of their different protocols rather than assuming that their success measures are interchangeable.