CogAgent: A Visual Language Model for GUI Agents

GUI screenshots contain small labels, icons, and spatial relationships that can disappear when an image is reduced to the resolution used by general vision-language models. Simply sending many more image patches through the language decoder increases computation. CogAgent addresses the architectural and training-data requirements of understanding these dense screens while retaining ordinary visual question-answering capabilities.

A dual-resolution architecture that combines global visual context with fine interface detail through a separate cross-attention pathway.

Continual pretraining on text recognition, visual grounding, and webpage-specific referring-expression tasks, followed by instruction tuning for GUI use.

Released CogAgent checkpoints and inference interfaces that support both screenshot interpretation and structured GUI action generation.

The screenshot enters two encoders. The existing CogVLM branch processes a smaller image and passes its visual tokens into the language decoder alongside the instruction. A smaller vision encoder processes a higher-resolution copy. Cross-attention in the decoder reads these detailed features without appending the entire high-resolution patch sequence to the main language sequence.

Training teaches the additional pathway to recognize text and associate descriptions with image regions. Webpage screenshots are paired with DOM-derived element descriptions and boxes for both description-to-location and location-to-description tasks. The DOM supplies supervision during data construction; it is not required as an observation when the trained GUI model receives a screenshot.

Instruction tuning combines visual question answering with annotated interfaces and converted action demonstrations. At use time, a prompt supplies the goal and action history with the current image; the model describes the intended operation and its target region. The original release provides a GUI-oriented agent checkpoint separately from a conversational checkpoint, so the selected model and prompt determine the output interface.

Visual question-answering benchmarks including VQAv2, TextVQA, DocVQA, and ChartQA test image, document, text, and chart understanding. Mind2Web and Android in the Wild test screenshot-conditioned action prediction, using benchmark-specific measures of element selection, operation correctness, and action matching.

Architecture and training-data ablations examine the contribution of the high-resolution branch and different pretraining sources. These GUI tests score actions against recorded benchmark states; they are distinct from an unrestricted live-computer completion test. This note covers the original CogAgent architecture rather than the later separately released CogAgent model.