Mind2Web: Towards a Generalist Agent for the Web

Web agents trained on a small collection of simplified sites may learn those interfaces without learning to operate unfamiliar websites. Real pages also contain far more HTML than a language model can conveniently process. Mind2Web addresses the need for diverse demonstrations, explicit generalization tests, and a practical way to select actions from large webpage representations.

A dataset of human-authored goals and browser demonstrations spanning varied real websites, domains, and interaction patterns.

Separate cross-task, cross-website, and cross-domain splits that distinguish different forms of generalization.

MindAct, a baseline combining a smaller element-ranking model with a larger language model for final action selection.

Released trajectory data, webpage records, candidate-element representations, and baseline resources supporting reproducible offline experiments.

Annotators propose a high-level task and demonstrate its completion through a browser tool. For each step, they identify the target element and select an operation, such as clicking, typing, or choosing an option. The collection process records webpage state and the interaction sequence; author verification aligns the task description with the retained actions and removes extraneous steps.

MindAct first represents individual HTML elements together with useful surrounding structure. A smaller ranking model scores each candidate against the task and previous actions. Only a selected candidate set proceeds to the larger prediction model, reducing the page information that must fit into its prompt while retaining candidate-specific context.

The prediction stage chooses the target element, operation, and any required value. Its multiple-choice formulation can process candidate groups before selecting a final action. The released dataset distinguishes annotated positive elements from other candidates and provides the cleaned representations used by the baseline. These records make it possible to inspect element retrieval separately from deciding what to do with the retrieved element.

The cross-task split holds out tasks on familiar websites; cross-website testing holds out sites within familiar domains; cross-domain testing holds out broader categories. Candidate ranking is evaluated by retrieval recall, while action prediction uses element accuracy, operation F1, step success, and an aggregate task-success measure.

Each evaluated step uses a cached webpage state and reference action history. Aggregate task success therefore reflects agreement across the recorded sequence, not autonomous completion in a live browser. The paper examines model choice and candidate filtering under this protocol and discusses how valid alternative action paths can be missed by a single recorded demonstration.