Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

A multimodal model can describe an image while struggling to identify exactly which region its words refer to. Numeric coordinates are awkward language outputs, and an unlabelled visual highlight offers no reusable name. Set-of-Mark studies whether explicit labels can connect visual regions to the model's textual response without updating its parameters.

A visual prompting scheme assigns speakable identifiers to image regions, allowing generated answers to refer back to a defined spatial partition.

An image-partition toolbox combines segmentation models and granularities, while mark-design procedures address label type, placement, and overlap.

A zero-shot evaluation suite applies the same prompting idea to region-level recognition, referring, segmentation, and phrase-grounding tasks.

An off-the-shelf segmentation system first proposes regions. The toolbox includes SEEM, SAM, and Semantic-SAM, supporting different levels of granularity and automatic or interactive partitioning. Each resulting region receives a visible identifier, such as a number, that is overlaid on the input image.

Prompting then asks the model to interpret the marked image. Depending on the task, the text may use ordinary language or explicitly refer to region identifiers. The model can identify a target by returning its label; the system maps that label back to the corresponding mask or region rather than requiring the model to reconstruct the geometry.

Mark placement is part of the input construction. Labels must remain distinguishable when regions overlap or have similar centers. The paper examines alternatives for locating and styling the marks, as well as which region proposals are supplied.

The segmentation system and the multimodal model retain separate roles: one defines candidate regions, and the other reasons about their meanings and relationships. No parameter update to GPT-4V is required for the prompting method.

Experiments use sampled validation examples from tasks including referring comprehension and segmentation on RefCOCOg, phrase grounding on Flickr30k, and generic segmentation. Predictions are interpreted through the corresponding region masks or boxes and evaluated with the task's spatial metric.

The original GPT-4V experiments use a fresh conversation for each example to avoid carrying context between tests. Comparisons include specialist vision models and other multimodal models. Ablations examine partition granularity and mark design; the study is not an end-to-end GUI-navigation evaluation.