AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents
Anchored State Memory stores intermediate task facts together with their evidence and dependency links.
Problem
Flat histories and free-form summaries can lose which entity a value belongs to, what supports it, and whether it remains valid.
Contributions
AndroTMem provides a long-horizon Android benchmark and an anchored representation for task-internal memory.
Method
Each memory anchor stores a type, content, evidence, and dependencies on earlier facts or subgoals.
The agent retrieves anchors relevant to the current observation and goal, then creates, revises, or invalidates them after acting.
The compared summary and anchor memories are generated by the agents rather than supplied as polished human annotations.
Evaluation
The benchmark contains 1,069 tasks and 34,473 steps, with completion judged through state anchors and their dependencies.
For Qwen2.5-VL-7B, the reported TCR is 42.39% with summaries and 46.19% with ASM; step-time improvements are not consistent across all models.
详细阅读笔记 · Y26-076
本笔记保留阅读时的论文版本、来源与实验边界。文中“你的方案／当前方案”等比较反映当时的讨论；当前研究方向见分类页。自拟例子与复现建议不是作者实验结果。
原论文 · 优先级 P0 · 复核日期：2026-09-07
与当前研究的关系：内容、来源、依赖、失效更新已存在，最关键历史基线。用户当前重心为同一任务的历史信息，跨任务经验另列条件，不能把两种记忆的收益混合。
0. 摘要
摘要完整译文（arXiv v1；CC BY 4.0；原摘要）。以下保留作者摘要的主张；译文按同一许可提供。
长程GUI代理是迈向真实部署的关键一步，但现有范式下的有效交互记忆仍缺乏研究。重放完整交互序列既冗余又放大噪声，而摘要常会抹去依赖关系所必需的信息和可追溯性。我们提出AndroTMem，一个用于长程Android GUI代理锚定记忆的诊断框架。其核心基准AndroTMem-Bench包含1069个任务、34473个交互步骤，平均每任务32.1步，最多65步。我们使用TCR（任务完成率）评估代理，重点考察需要向后续步骤传递关键中间状态才能完成的任务；AndroTMem-Bench旨在建立较强的步骤间因果依赖，使稀疏但关键的中间状态成为后续动作的决定因素，并将交互记忆置于评估中心。在开源和闭源GUI代理中，我们观察到一致模式：随着交互序列增长，性能下降主要由任务内部的记忆失败引起，而非孤立的感知错误或局部动作失误。基于这一诊断，我们提出锚定状态记忆（ASM），用一组紧凑、具有因果联系的中间状态锚点表示交互序列，支持面向子目标的检索和具有归因意识的决策。在多种设置及12个被评估GUI代理上，ASM持续超过完整序列重放和摘要基线，将TCR提高5%–30.16%、AMS提高4.93%–24.66%，表明锚定的结构化记忆能够有效缓解长程GUI任务的交互记忆瓶颈。代码、基准和相关资源已在此处公开。
1. 方法动机
跨应用任务经常先读到某个价格、地址或联系人，若干步后才真正使用。这个信息可能既不在初始指令里，也无法从当前页面恢复；只保留最近截图会丢失它。
自由摘要虽然短，却容易混淆数值归属，或把尚未验证的操作当作已经完成。论文的直觉是将历史压缩成带证据、可关联的中间事实，而不是简单缩写操作流水账。
2. 方法设计
§3.1 Long-Horizon Task Formulation
构造具有真实依赖的长任务。 将50个常用应用按功能分组，使用70多个跨应用模板组织需求，再由模型改写为自然表达。模板不预先告诉代理后续需要使用的所有中间结果，迫使它从交互中取得并记住。
§3.2 Data Pipeline
同步采集轨迹与状态锚点。 标注者逐步查看截图和同步可访问性信息，记录动作、推理、摘要以及关键中间状态，再执行到下一页面。任务和轨迹分别进行质量检查，保证状态证据与步骤对齐。
§5.2 Anchored State Memory Definition
将记忆表示成有来源的状态对象。 每个锚点包含类型、内容、证据和依赖链接，可表示抽取数值、识别实体、已完成子目标、持久变化或待处理异常。链接说明某个结论由哪些先前事实推导而来。
§5.2 锚点读取
按当前观察检索相关锚点。 执行时从锚点库选取与当前界面和用户目标有关的子集，与当前页面一起用于决策。这样跨应用得到的事实可以在需要时重新出现，而无需携带所有旧截图。
§5.2 状态更新
行动后更新而非只追加。 分析当前步骤是否产生新状态、是否应修正或使旧锚点失效，以及是否形成新依赖。论文的状态对象支持这些操作，但具体检索排序和各类失效规则未在主体中完全算法化。
§5.3 History Utilization Ablation
让模型自行生成受比较的记忆。 消融中摘要和ASM都由各代理依据统一规则自动从历史生成，不直接向模型提供人工精制锚点。评估同时检查动作匹配，以及是否在满足前序因果依赖的条件下达到最终锚点。
自拟例子：把记忆在什么时候使用讲清楚
在地图中读到诊所地址，再在打车应用输入。锚点关联诊所名、地址、来源步和后续用处。若两个诊所同名，只有一个文字值可能不够，原区域的上下文可用于消歧。
模型输入、输出与记忆边界
| 项目 | 本文实际范围 |
|---|---|
| 记忆来源 | 同任务锚定状态证据 |
| 与本方案相同或不同 | 内容、来源、依赖、失效更新已存在，最关键历史基线 |
| 应固定的对照条件 | 用同预算ASM与ASM＋视觉回查测摘要无法保真的细节；不能把加时间戳称首次。 |
训练、模型规模、具体输入输出及调用次数见各方法小节；没有查明的参数不填入建议值冒充原文默认值。
3. 与其他方法对比
| 对比对象 | ASM的变化 | 收益与代价 |
|---|---|---|
| 原始截图—动作历史 | 只保留决策关键状态 | 减少输入干扰，依赖状态抽取正确 |
| 粗粒度自由摘要 | 显式记录证据和因果链接 | 较少遗漏依赖，结构化记录可能更长 |
| 仅保存已执行动作 | 区分操作和已验证结果 | 防止虚假进度，需要观察结果 |
| 固定追加记忆 | 支持更新和失效 | 能处理变化，但失效判断仍可能出错 |
贡献在于用可复用的中间状态表示长期依赖。它没有把滑动过程编译成免模型调用的循环。
4. 实验表现与优势
数据集含1,069个任务、34,473步，平均32.1步、最多65步。动作匹配中点击采用相对距离或目标分割区域，滑动简化为方向匹配；TCR以状态锚点及其依赖满足情况判定，不能直接等同于任意生产系统的真实业务成功率。§3.3、§4.1
表4中，Qwen2.5-VL-7B原始历史每步8.102秒、TCR 16.04%；摘要为2.922秒、42.39%；ASM为2.511秒、46.19%。但不同模型并非一致更快：Gemini-3-Flash的ASM每步8.345秒，摘要5.427秒，原始历史8.108秒。表4
因此论文支持结构化记忆提高所测正确性，但“总是比摘要更省时间”不成立。所有上述时间均为平均每步口径；完整任务耗时还受步数、失败和跨应用等待影响。
5. 学习与应用
论文提供代码与数据入口，本轮未读取实现。最容易复用的对象是四字段锚点及因果链接，而非某个未经说明的固定摘要长度。
复现建议：先选择必须跨应用搬运数值的任务，保存“值—对象—证据步骤—依赖”的记录；比较原始历史、同预算摘要和ASM。额外加入数值被修改、同名实体及操作失败的样本，观察锚点更新是否及时，并把记忆生成成本计入每步时间。
6. 总结
核心思想：记住带证据和依赖的中间状态。
速记流程：完成一步操作→提取关键结果→关联来源与依赖→需要时检索并更新
来源与核查：原论文，已读§3、§4.1、§5及表4；资源链接取自论文。
对“历史摘要＋局部视觉证据回查”的具体启发
用同预算ASM与ASM＋视觉回查测摘要无法保真的细节；不能把加时间戳称首次。
不能把检索到的旧图片直接当作当前可点击坐标。历史图可证明过去出现过什么；当前动作的位置和状态必须由当前观察核验。研究评估需区分过去事实回忆正确、当前状态有效和最终动作正确三个环节。
原文方法图
原文图示与图题。下面直接引用作者图像；解释以上文为准。
原表核查附录
以下保留原表数值、行序与英文方法名称，便于核查；标题与分组行可能在HTML中跨列。各指标含义、测试划分和可比较条件见上面的实验解释，空格不等于0。
原表 1（点击展开）
| Dataset | #Traj | #Steps | #App | GT | Low-Level Instr. | History Modeling | History Modeling |
|---|---|---|---|---|---|---|---|
| Summary | ASM | ||||||
| AMEX Chai et al. [2025] | 2,946 | 12.8 | 110 | ✓ | ✗ | ✗ | ✗ |
| AITW Rawles et al. [2023] | 30,378 | 6.5 | 357 | ✓ | ✗ | ✗ | ✗ |
| AITZ Zhang et al. [2024b] | 2,504 | 7.5 | 70 | ✓ | ✓ | ✓ | ✗ |
| Android Control Li et al. [2024] | 15,283 | 5.5 | 833 | ✓ | ✓ | ✓ | ✗ |
| Android World Rawles et al. [2024] | 116 | – | 20 | ✗ | ✗ | ✗ | ✗ |
| Mobile-R1 Gu et al. [2025a] | 4,635 | 5.3 | 28 | ✓ | ✓ | ✗ | ✗ |
| Mobile-Bench-v2 Xu et al. [2025a] | 12,856 | 7.28 | 49 | ✗ | ✗ | ✗ | ✗ |
| GUIOdyssey Lu et al. [2025a] | 8,334 | 15.3 | 212 | ✗ | ✓ | ✓ | ✗ |
| SPA-Bench Chen et al. [2024b] | 201 | 8.2 | 21 | ✗ | ✗ | ✗ | ✗ |
| LearnGUI Liu et al. [2025a] | 2,353 | 13.2 | 73 | ✓ | ✓ | ✓ | ✗ |
| LongHorizonUI Kang et al. [2026] | 354 | 22.1 | 28 | ✓ | ✓ | ✓ | ✗ |
| AndroTMem-Bench | 1,069 | 32.1 | 50 | ✓ | ✓ | ✓ | ✓ |
原表 2（点击展开）
| Model | AMS | TCR |
|---|---|---|
| Closed-Source | ||
| GPT-4o Hurst et al. [2024] | 14.24 | 11.75 |
| GPT-5 Singh et al. [2025] | 12.37 | 11.46 |
| Gemini-2.5-Flash Comanici et al. [2025] | 15.62 | 14.51 |
| Gemini-2.5-Pro Comanici et al. [2025] | 32.71 | 41.11 |
| Gemini-3-Flash | 46.14 | 55.21 |
| Open-Source | ||
| Qwen2.5-VL-7B Bai et al. [2025] | 20.71 | 16.04 |
| UI-TARS-1.5-7B Qin et al. [2025] | 35.62 | 34.55 |
| AgentCPM-GUI-8B Zhang et al. [2025] | 14.06 | 10.49 |
| UI-Venus-Navi-7B Gu et al. [2025b] | 14.08 | 9.64 |
| InfiGUI-R1 Liu et al. [2025e] | 14.12 | 9.20 |
| Multi-Agent | ||
| Mobile-Agent-E Wang et al. [2025a] | 15.77 | 11.71 |
| COLA Zhao et al. [2025] | 17.22 | 11.85 |
原表 5（点击展开）
| Category | Applications | Description |
|---|---|---|
| Shopping | JD, Taobao, Pinduoduo, Suning | Product browsing, price comparison, order placement, and online payment for consumer goods. |
| Travel | Ctrip, Didi Chuxing, Fliggy | Trip planning, transportation booking, ride-hailing services, and real-time travel management. |
| Map | Amap, Baidu Maps, Tencent Maps | Location search, route planning, navigation, and real-time traffic information. |
| Gmail, Outlook | Email communication, message management, information exchange, and productivity-oriented correspondence handling. | |
| Cloud Storage | Baidu Netdisk, Google Drive | Cloud-based file storage, synchronization, backup, and cross-device file sharing. |
| Meeting | DingTalk, Tencent Meeting, WeCom, Zoom | Online meetings, video conferencing, real-time communication, and collaborative work coordination. |
| Note-taking & Document Tools | Feishu (Lark), WPS Office, DingTalk, Notes, Youdao Notes | Document editing, note-taking, information organization, and collaborative content creation. |
| Social | WeChat, QQ, Weibo, Xiaohongshu | Social networking, instant messaging, content sharing, and online community interaction. |
| Category | Applications | Description |
| Entertainment | Bilibili, Douyin, Tencent Video, iQIYI, YouTube | Consumption of short-form and long-form video content, live streaming, and entertainment media browsing. |
| News & Information | Toutiao, NetEase News, Sina News | News aggregation, personalized information feeds, and real-time information consumption. |
| System Tool | File Manager, Gallery, Settings, Browser, Recorder, Calculator, Clock, Calendar, Chrome | Device management, system configuration, file organization, and basic utility functions. |
| Knowledge | Zhihu | Knowledge sharing, question answering, expert content consumption, and community-based discussions. |
| Music App | NetEase Cloud Music, QQ Music, Kugou Music, Kuwo Music | Music streaming, playlist management, audio search, and personalized music recommendation. |
| Takeout | Meituan, Ele.me | Food delivery services, restaurant browsing, order placement, and logistics tracking. |
| Payment | Alipay | Mobile payment, financial transactions, digital wallet management, and online-offline payment integration. |
| Health | Keep | Fitness training, exercise tracking, health monitoring, and personalized workout guidance. |
原表 6（点击展开）
| Task Type | Primary Intent | Example Scenarios |
|---|---|---|
| Lookup | Information retrieval/verification | Checking shop hours, searching for news, or finding answers to specific questions. |
| Compare & Decide | Decision making via comparison | Comparing menus of three restaurants or checking product parameters across different platforms. |
| Purchase / Order | Transaction completion | Placing food delivery orders, clearing a shopping cart, or purchasing memberships. |
| Booking | Resource/Time reservation | Booking hotel rooms, reserving train tickets, or scheduling a doctor’s appointment. |
| Communicate | Coordination with others | Confirming meeting details with a contact or initiating a group discussion. |
| Share / Recommend | Content distribution | Sending a store link to a friend or sharing news and music playlists to social media. |
| Create Content | Generative output | Writing a note, editing a document, or posting a review/comment on a platform. |
| Configure | System/App adjustment | Granting location permissions, enabling notifications, or modifying system settings. |
| Payment | Financial processing | Paying for an existing order, applying for a refund, or processing utility bills. |
| Navigation | Trip execution | Planning a route between locations or hailing a ride and sharing the trip status. |
原表 7（点击展开）
| Action | Argument | Functionality |
|---|---|---|
| TAP | [pos1] | click the on-screen position |
| INPUT_TEXT | [text] | input the text into the currently focused text field using the system keyboard |
| LONG_PRESS | [pos1] | press the screen for a long time to copy texts or download images |
| SWIPE | [pos1] | swipe the screen in a given direction (up, down, left, or right) to scroll the page |
| SWIPE_TWO_POINT | [pos1,pos2] | swipe from position pos1 to position pos2, enabling precise scrolling or drag-and-drop operations |
| WAIT | – | wait for a specified duration or until the UI state changes |
| FINISH | – | the sign that the instruction has been completed |
| OPEN_APP | – | launch the specified application |
| CAPTURE_SCREEN | – | capture the current screen for visual perception or state analysis |
| HOME | – | go to the home screen |
| BACK | – | go to the previous screen |
原表 8（点击展开）
| Type Name | Recommended Values | Description |
|---|---|---|
| Key Subgoal Achievement | SUBGOAL | Completion of intermediate tasks, such as successful login, retrieval of search results, or adding an item to the shopping cart |
| State Transition and Mode Switching | STATE_CHANGE | Changes in the environment or application mode, such as entering the payment interface or opening a permission request page |
| Causally Dependent Steps | DEPENDENCY | Steps whose execution depends on prior data or states, such as copying text or filling in a form |
| Exception Handling | EXCEPTION | Actions taken to address interruptions or errors, such as closing pop-up advertisements or retrying a network request |
| Global Constraint Information | CONTEXT_INFO | Essential contextual information that constrains task execution, such as selecting dates or setting filter conditions |
| Task Completion Indicator | FINISH | The sign that the instruction has been completed |
Open this note in the interactive notebook (comments, hooks) → · All notes