MGA: Memory-Driven GUI Agent for Observation-Centric Interaction
MGA separates screen description, action-outcome verification, planning, and coordinate grounding.
Problem
Goal-focused descriptions can omit useful interface content, and issuing an action does not establish that it succeeded.
Contributions
The architecture adds task-independent structured observations and incrementally verified interaction memory.
Method
A fine-tuned observation model describes the current screenshot, including non-interactive text and semantic state.
A memory model compares consecutive observations and the executed action to record success, failure, uncertainty, and state changes.
A planner uses the current image and verified history to propose a semantic action, which a grounding model maps onto the current screen.
Evaluation
The reviewed v3 reports OSWorld-based system comparisons and ablations removing observation or memory support.
Smaller memory models reduce the reported cost but also lower performance; the note retains the paper's ambiguity between a grounding-labelled table and terminal-state evaluation.
详细阅读笔记 · P070
本笔记保留阅读时的论文版本、来源与实验边界。文中“你的方案／当前方案”等比较反映当时的讨论；当前研究方向见分类页。自拟例子与复现建议不是作者实验结果。
目录 / Contents
原论文 · 优先级 P0 · 复核日期：2026-09-07
同任务动作验证、结构化状态与增量历史记忆。与研究方案的关系：小观察模型＋小记忆模型给主规划器供信息，与你的架构近；未提供历史原图RAG。
0. 摘要
中文转述，非逐句翻译。MGA把观察、记忆、规划和定位分开。观察模型先完整描述当前页面，记忆模型比较动作前后状态并判断执行结果，规划器读取这些结构化信息选择下一步，再由定位器把操作目标落到坐标。记忆重在让主模型知道真实发生了什么。
1. 方法动机
仅用目标导向的视觉描述容易忽略只读文本、背景信息或尚未成为目标的状态。另一方面，代理不能把“已经发出点击”直接等同“操作成功”。作者以任务无关观察和经过验证的历史，降低这两类错误。
2. 按原文小节解释方法
原文§3.1 Problem Formulation
当前状态由当前截图、空间语义表示及先前验证记忆构成。当前图仍保留给决策，不能把额外描述理解为完全替代视觉输入。系统划分四个模型角色，目的在于明确信息流，而不是四个独立行动代理同时操作。
原文§3.2 Observation Agent
Qwen2.5-VL-7B经GUI数据微调，生成尽量任务无关的界面表示，覆盖窗口结构、文本内容及语义状态，包括非交互元素。这样即使当前目标没提某个数字，数字也有机会进入状态描述。它仍由生成模型筛选内容，不保证像素级完备。
原文§3.3 Memory Agent
记忆器比较前后观察和已执行动作，把结果分为成功、失败或不确定，再记录动作、判断和状态变化。连续失败、重复动作或停滞形成异常信号。历史是追加的交互记录，不能据此宣称有严格固定大小的上下文。
原文§3.4 Planner Agent
主规划器接收任务、当前视觉观察、结构化页面和已验证历史，决定下一目标；例如先前点击保存没有发生预期变化，就重试或换路线。记忆器只报告结果，不替代规划器发出新任务动作。
原文§3.5 Grounding Agent
定位器根据当前屏幕把语义动作转为实际坐标并执行。即使历史记住某按钮曾在右上角，也应以当前图重新定位。论文使用UI-TARS等定位组件，不能把记忆中的旧坐标直接作为当前坐标。
自拟例子：保存联系人后核验
点击保存后页面仍停留在编辑态，并出现格式错误提示。观察器记录提示，记忆器把本次保存标为失败，规划器修正号码。若几步后需要核查原号码而摘要已覆盖它，MGA现有文字记忆仍可能不够，你的视觉证据回查可以专门处理这类遗失。
3. 模型输入输出与训练/推理分工
观察器：当前图→结构化页面；记忆器：前后状态＋上一动作＋旧记忆→验证结果与更新历史；规划器：任务＋当前图＋页面＋历史→语义操作；定位器：操作＋当前图→坐标。当前v3实验包含GPT-5规划器、Qwen3-8B或GPT-5记忆器及UI-TARS定位器。
观察模型有微调，不能统称全系统训练免费；其余主要采用预训练角色提示。小记忆模型可降低费用，但需同时测摘要质量。当前核查为2026-04修订v3，稿件写ACM MM投稿信息，不是录用证明。
4. 实验、指标和原表解释
主表MGA GPT-5总体64.7，OS-Symphony63.6。原表标题存在grounding accuracy表述，而实验协议描述OSWorld终态规则检查，因此本卡称原表总体分数并保留协议，避免自行把不同指标合并。工作流比较中，Qwen3记忆器47.7、报告费用$15，GPT-5记忆器56.3/$64，OS-Symphony54.9/$150；费用严格沿用作者实验口径，不外推为普通API单步价格。
5. 消融、附录与证据限制
工作流消融完整56.3，去观察46.0，去记忆39.0，两者都去36.8，说明两种补充信息各有作用。较小记忆器虽省费但结果明显降低，直接回应“3B概括一定足够吗”：必须验证。附录补充提示和成本；原表总体成绩不能隔离历史压缩本身，因为观察与规划也改变。
6. 对当前历史信息方案的比较与复现建议
P0阅读。你的原始模块排列与其接近，应把研究变量集中在历史事实被压缩后能否恢复；采用相同观察器和规划器，比较MGA文本增量记忆、纯摘要和有来源的视觉回查。不要把“分一个小模型概括”单独当创新。
历史证据表明过去发生过什么，不能直接证明当前页面还保持同一状态；本方案应分别验证检索内容、来源归属和状态时效。
核查原文；已读：§3.1–3.5、§4.1–4.3、主结果及工作流/组件消融表、附录提示与成本。未阅读实现源码，资源链接存在不等于确认可完整复现。
原文方法图
原文图示与图题。下面直接引用作者图像；解释以上文为准。
原表核查附录
以下保留原表数值、行序与英文方法名称，便于核查；标题与分组行可能在HTML中跨列。各指标含义、测试划分和可比较条件见上面的实验解释，空格不等于0。
原表 1（点击展开）
| Framwork | Steps | Office | Daily | Professional | OS | Workflow | Overall |
|---|---|---|---|---|---|---|---|
| General Models & Specialist Model | |||||||
| O3 | 50 | 9.4 | 20.5 | 30.6 | 37.5 | 11.8 | 17.1 |
| OpenCUA-32b | 50 | 29.9 | 40.9 | 65.3 | 60.9 | 14.6 | 34.1 |
| OpenCUA-72B | 100 | 44.7 | 49.9 | 72.5 | 61.1 | 22.1 | 44.9 |
| UI-TARS | 100 | 50.4 | 55.6 | 51.0 | 41.6 | 14.6 | 41.8 |
| UI-TARS-2 | 100 | 61.1 | 62.1 | 61.2 | 41.6 | 34.1 | 53.1 |
| DeepMiner-Mano-7B | 100 | 39.2 | 44.8 | 73.4 | 50.0 | 17.2 | 40.1 |
| Claude-Sonnet-4.5 | 50 | 62.4 | 57.6 | 63.2 | 70.8 | 47.0 | 58.1 |
| Qwen3-VL-32B-Instruct | 50 | – | – | – | – | – | 32.4 |
| Qwen3-VL-32B-Thinking | 50 | – | – | – | – | – | 41.0 |
| Agentic Framework | |||||||
| Agent S3 w/ GPT-5-Mini | 50 | 54.6 | 46.6 | 44.9 | 62.5 | 37.0 | 47.5 |
| UiPath w/ GPT-5 | 50 | 49.5 | 62.1 | 71.4 | 73.9 | 37.3 | 53.7 |
| Jedi-7B w/ o3 | 50 | 47.0 | 62.1 | 69.4 | 54.2 | 34.9 | 50.6 |
| CoAct-1 w/ GPT-5 | 100 | 62.9 | 57.9 | 71.4 | 75.0 | 47.8 | 59.9 |
| CoAct-1 w/ GPT-5 | 150 | 62.9 | 61.7 | 71.4 | 75.0 | 47.8 | 60.8 |
| GTA1-7b w/ o3 | 50 | 46.1 | 44.9 | 77.6 | 58.3 | 37.1 | 48.6 |
| GTA1-7b w/ o3 | 100 | 54.6 | 60.3 | 61.2 | 62.5 | 38.3 | 53.1 |
| GTA1-7b w/ GPT-5 | 50 | - | - | - | - | - | 61.0 |
| OS-SYMPHONY w/ Qwen3-VL-32B | 50 | 40.9 | 53.5 | 75.1 | 58.3 | 31.2 | 46.9 |
| OS-SYMPHONY w/ GPT5-Mini | 50 | 58.2 | 61.4 | 75.0 | 73.7 | 47.4 | 58.1 |
| OS-SYMPHONY w/ GPT-5 | 50 | 64.9 | 61.2 | 69.2 | 75.0 | 54.9 | 63.6 |
| Our Methods: MGA | |||||||
| MGA w/ o3 | 50 | 49.0 | 62.8 | 70.6 | 68.2 | 36.4 | 52.9 |
| MGA w/ GPT-5 | 50 | 64.7 | 64.4 | 85.4 | 87.5 | 47.7 | 64.7 |
原表 2（点击展开）
| Framework | Workflow SR (%) | Cost ($) |
|---|---|---|
| OS-Symphony | 54.9 | 150 |
| MGA ( w/ Qwen3 memory ) | 47.7 | 15 |
| MGA ( w/ GPT-5 memory ) | 56.3 | 64 |
原表 3（点击展开）
| Configuration | GPT-5 | O3 |
|---|---|---|
| MGA (Full) | 56.3 | 36.4 |
| - w/o obs | 46.0 | 35.1 |
| - w/o memory | 39.0 | 27.7 |
| - w/o obs + w/o memory | 36.8 | 15.8 |
原表 4（点击展开）
| Action | Parameters | Description | General Constraints |
|---|---|---|---|
| Semantic-grounded GUI Actions | |||
| click | instruction: str num_clicks: int button_type: str hold_keys: list | Click on a semantically described UI element. | Target elements are specified via semantic descriptions. |
| drag_and_drop | starting_desc: str ending_desc: str hold_keys: list | Drag from one semantically described element to another. | Both source and target are resolved via semantic grounding. |
| type | element_desc: str text: str overwrite: bool enter: bool | Type text into the specified UI element. | Supports optional overwrite and submission behavior. |
| hotkey | keys: list | Execute a keyboard shortcut. | Keys are provided as standard key combinations. |
| scroll | instruction: str clicks: int shift: bool | Scroll within a semantically specified region. | Scroll direction and magnitude are parameterized. |
| Programmatic Execution via code | |||
| code | code: str | Execute a Python script for direct system-level operations. | Enables precise manipulation of structured data and background resources. |
| Control and Termination Signals | |||
| wait | time: float | Pause execution for a specified duration. | Handles asynchronous UI responses and latency. |
| done | return_value: Optional | Signal successful task completion. | Terminates execution and optionally returns results. |
| fail | None | Signal unsuccessful termination. | Used when the task cannot be completed under the current trajectory. |
原表 5（点击展开）
| Phase | Steps | Memory State Update | Consequent Agent Action |
|---|---|---|---|
| 1. Knowledge Correction | 1–8 | Action verification detects “Folder is empty”, correcting the knowledge base: files reside on the Desktop rather than the target folder. | Aborts folder search and redirects grounding toward Desktop icons via the open dialog. |
| 2. Working Memory | 9–23 | Aggregates fragmented options from scrolling interactions and stores derived answers (e.g., Test 2 = “baaad”, Test 3 = “aaaaa”). | Returns to Answer.docx using pre-computed semantic strings, avoiding redundant copy-paste operations. |
| 3. Anti-Stagnation | 24–27 | Stagnation detection triggers after repeated click and enter fail to produce visible changes. | Terminates ineffective retries and applies a fallback strategy ( End hotkey) to reset cursor position. |
| 4. Loop Avoidance | 28–33 | Action verification returns UNCERTAIN due to lack of visual feedback during save operations and flags repeated hotkey usage. | Prioritizes global task completion over local uncertainty and safely terminates via done() . |
原表 6（点击展开）
| Framework | Avg. Step | Prompt Tokens | Completion Tokens | Total |
|---|---|---|---|---|
| OS-Symphony | 15.2 | 572.9K | 25.7K | 598.6K |
| MGA | 19.3 | 188.8K | 37.3K | 226.1K |
原表 4（点击展开）
| click (desc, n, type, keys) | Click described element. n=2 (double) for opening files; n=3 for paragraphs. |
|---|---|
| type (desc, text, ovr, ent) | Type text. overwrite=True clears field; enter=True submits. |
| drag_and_drop (start, end) | Drag from start description to end description. No coordinates. |
| code (description) | Execute natural language logic as Python scripts (Calculations/Excel batch). |
| hotkey (keys) | scroll (desc, n) | Execute combinations (e.g., [’ctrl’, ’s’]) or scroll within elements. |
| done () | fail () | Signal task completion or trigger replanning on failure/stagnation. |
Open this note in the interactive notebook (comments, hooks) → · All notes