← 返回阅读笔记 / Back to reading note

LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

SUP-A04 原表

来源:作者原文。2026-09-07抓取,原始行列,不是统一复跑成绩。

Table 1: Comparison with existing memory and long-context benchmarks. # Sess., # Tok., and # Q denote the history size in sessions or tokens, as well as the total number of questions. MM denotes whether the context or question is a multimodal mixture of text and images. For benchmarks with multiple preset length tiers, we report a range between the minimum length tier and the maximum length tier. For the other benchmarks, we report the averaged context size over all examples.
Benchmark Domain Context Profile Question Profile Memory Ability
# Sess. # Tok. MM # Q MM Static Dynamic Workflows Gotchas Premise
General Long Context
LongBench V2 Mixed N/A 260k 503
MemoryAgentBench Mixed N/A 285k 2,071
CL-Bench Mixed N/A 10k 1,899
Conversational Long Context
LoCoMo User-user chat 28 \sim16k 7,512
LongMemEval-V1 User-assistant chat 48–475 115k–1.5M 500
PersonaMem User-assistant chat 5–60 26k–951k 5,990
PersonaMem-v2 User-assistant chat 10–20 33k–124k 5,000
BEAM User-assistant chat 4.5–100 124k–10M 2,000
Agentic Long Context
MemoryArena Agent (mixed) 7 40k+ 766
AgentLongBench Game agent 1 31k–4M 6,400
EMemBench Game agent 1 2k–\infty 1,280+
FileGramBench File-system agent 12 11k 4,333
AMA-Bench Agent (mixed) 1 57k 2,496
LongMemEval-V2 Web agent 100–498 25M–115M 451
Figure 4 : Pilot studies on LME-V2 non-abstention questions. (a) Frontier LLMs perform poorly without trajectory history, suggesting that parametric knowledge alone is insufficient for LME-V2. (b) LME-V2 is challenging to answer even with oracle answer-bearing trajectories and optimizations such as evidence slicing with notes or using a coding agent harness help improve performance.
Context Overall Static Dynamic Workflow Gotchas
Qwen3.5-9B (thinking enabled)
No context 0.016 0.000 0.015 0.155 0.136
Oracle trajectories 0.596 0.566 0.668 0.718 0.310
Oracle slices + notes 0.825 0.908 0.879 0.750 0.484
GPT-5.4-mini (medium reasoning)
No context 0.045 0.025 0.010 0.075 0.171
Oracle trajectories 0.653 0.660 0.696 0.697 0.484
Oracle slices + notes 0.863 0.950 0.843 0.905 0.467
Codex + GPT-5.4-mini (xhigh reasoning)
Oracle trajectory files 0.897 0.986 0.947 0.815 0.517
Table 2: Main results with baselines and ablations. The downstream reader is always Qwen3.5-9B. We boldface the best results in each method family. ✣ means statistically significantly outperforming the non-ablation baselines via paired bootstrap test ( p < 0.05 p<0.05 ). AgentRunbook strongly outperforms the baseline in RAG family and achieves a superior latency in coding agent family.
Method LME-V2-Small LME-V2-Medium
Overall Static Dynamic Workflow Gotchas Latency Overall Static Dynamic Workflow Gotchas Latency
No retrieval 0.013 0.000 0.008 0.094 0.138 0s 0.013 0.000 0.008 0.094 0.138 0s
RAG Methods (Controller = Qwen3.5 9B, thinking enabled)
RAG: query \rightarrow slice 0.428 0.471 0.425 0.415 0.207 0.1s 0.381 0.434 0.405 0.293 0.242 0.1s
RAG: query \rightarrow slice + notes 0.510 0.524 0.496 0.528 0.414 0.2s 0.459 0.487 0.472 0.434 0.310 0.3s
AgentRunbook-R 0.586 0.661 0.583 0.528 0.310 26.9s 0.570 0.630 0.614 0.472 0.345 25.8s
        – raw slice pool 0.423 0.286 0.551 0.538 0.345 16.7s 0.335 0.233 0.433 0.377 0.413 17.1s
        – event pool 0.556 0.614 0.559 0.528 0.276 19.1s 0.484 0.534 0.496 0.434 0.276 18.5s
        – note pool 0.579 0.651 0.614 0.481 0.310 22.8s 0.499 0.561 0.543 0.396 0.276 20.5s
Coding Agent Methods (Controller = GPT-5.4-mini, xhigh reasoning)
Codex 0.699 0.804 0.670 0.575 0.586 177.2s 0.687 0.783 0.646 0.613 0.517 185.8s
AgentRunbook-C 0.749 0.820 0.724 0.726 0.483 108.3s 0.701 0.788 0.701 0.613 0.449 139.9s
        – workflow 0.701 0.772 0.677 0.632 0.586 167.9s 0.641 0.709 0.646 0.575 0.414 231.9s
        – manifest artifacts 0.747 0.847 0.709 0.698 0.448 155.0s 0.681 0.767 0.685 0.576 0.483 211.6s
        – helper functions 0.714 0.783 0.724 0.660 0.414 145.9s 0.718 0.804 0.693 0.689 0.380 182.5s
Table 3: Examples of goal sanitization for WorkArena and WorkArena++ Level-2/3 trajectories. The rewritten goals preserve task intent and task-specific values while removing explicit navigation routes and step-by-step module hints.
Task family Original goal excerpt Sanitized goal excerpt
Duplicate problem cleanup Clean-up your duplicate problems. Concretely, navigate to the “Assigned to me” module of the “Problem” application. Create a filter where “Problem statement” contains #SERIES-5ea261ef-8. Mark problems with duplicated problem statements as such. Clean-up your duplicate problems. Review your own assigned problems where “Problem statement” contains #SERIES-5ea261ef-8. Mark problems with duplicated problem statements as such.
Dashboard-driven catalog restocking Retrieve information from the chart with title #CAT012007808. Navigate to Reports > View/Run, search for the report, then navigate to Self-Service > Service Catalog and place an order for the least available item. Retrieve information from the chart with title #CAT012007808. Find the greatest stock value and the least available item in the chart. For the least available item, place an order for extra items such that its quantity matches the value you found.
Requested-item reorder Order same item as Kathryn-Lisa Ibarra-Stewart. Navigate to the “Requested Items” module of “Self-Service”, filter by “Requested for”, then navigate to the “Service Catalog” module and order the item with the specified quantity and configuration. Order same item as Kathryn-Lisa Ibarra-Stewart. Find the item previously requested for Kathryn-Lisa Ibarra-Stewart. Order the item with the specified quantity and configuration.
Hardware asset lookup Find the warranty expiration date for Julia-Dylan Ray-Mclean’s laptop. Navigate to Portfolios > Hardware Assets, filter where “Assigned to” is Julia-Dylan Ray-Mclean, and extract the “Warranty expiration” field. Find the warranty expiration date for Julia-Dylan Ray-Mclean’s laptop and report it.
Table 4: Reader prompt used in the main context gathering evaluation. The same template is used for all memory systems.
Component Template
System prompt, WebArena You are an experienced colleague in a web browsing environment that has a customized Magento-based shopping website, a customized Magento-based shopping admin CMS website, as well as a customized forum website based on Reddit/Postmill. Answer based on your memory of the environment. If you do not know the answer, output exactly \boxed{UNKNOWN}. Do not guess. Never attempt to guess an answer if you are not sure. If you believe the question’s construction/premise is wrong, provide an explanation in \boxed{} explaining why the question is flawed.
System prompt, WorkArena You are an experienced colleague working in a customized ServiceNow environment. Answer based on your memory of the environment. If you do not know the answer, output exactly \boxed{UNKNOWN}. Do not guess. Never attempt to guess an answer if you are not sure. If you believe the question’s construction/premise is wrong, provide an explanation in \boxed{} explaining why the question is flawed.
User prompt ### Memory context:
{memory context returned by the memory module, after truncation}
### Question to answer:
{question text}
If the question contains an image, the image is appended after the question text.
Table 5: LLM-judge prompts for semantic evaluation of abstention and gotchas questions. Both judges return JSON with a binary label.
Judge Prompt
Abstention judge System: You are a strict grader for flawed-premise (abstention) questions. Judge whether a model answer correctly identifies that the question premise is wrong, consistent with the reference answer. If the model follows the flawed premise and gives a concrete answer under that premise, it must be graded 0. If the model’s final answer is just UNKNOWN / cannot determine without identifying the flaw, grade 0. If the model is contradictory (both rejects premise and also gives a concrete premise-following answer), grade 0. Paraphrases are allowed when they preserve the same core flaw described by the reference answer. User: Evaluate whether the model correctly handled a flawed-premise question. Question: {question_text} Reference answer: {reference_answer} Model full response: {model_full_response} Model extracted final answer: {model_final_answer} Scoring rubric: Label 1 only if the model clearly identifies the flawed premise and reaches a conclusion equivalent to the reference answer. Label 1 also if the model explicitly says it lacks access to the user’s specific live environment/instance/configuration and therefore cannot verify, provided it does not give a concrete premise-following answer. Label 0 if the model follows the flawed premise and gives a concrete answer under that premise. Label 0 for generic UNKNOWN/insufficient-info replies that do not identify a flaw and do not make the explicit environment-access limitation clear. Label 0 if contradictory. Output JSON only: {"label": 0 or 1, "reason": "short rationale"}
Gotchas judge System: You are a strict grader for gotchas-style insight questions. The reference answer describes the key insight(s). Grade 1 if the model response includes at least one correct insight point from the reference answer (paraphrase allowed), and does not contradict any reference point. If the model’s direction is wrong, or it contains contradictions against any reference point, grade 0. If the model gives multiple points, partial coverage is enough for 1 as long as no contradictions appear. User: Evaluate whether the model answer captures the gotcha insight. Question: {question_text} Reference answer: {reference_answer} Model full response: {model_full_response} Model extracted final answer: {model_final_answer} Scoring rubric: Label 1 if the model includes at least one correct insight point from the reference answer (paraphrase acceptable), and does not contradict any reference point. Label 1 even if only part of a multi-point reference answer is covered, as long as there is no contradiction. Label 0 if direction is wrong (suggests opposite action/cause), even if some wording overlaps. Label 0 if any point in the model response contradicts any reference point. Label 0 if the response is irrelevant or generic without insight. Output JSON only: {"label": 0 or 1, "reason": "short rationale"}
Table 6: No-context direct QA results on non-abstention questions. Frontier LLMs perform poorly across problem types.
Model Overall Static Dynamic Workflow Gotchas
GPT-5.2 0.047 0.000 0.000 0.032 0.210
Gemini-3.1-Pro-Preview 0.110 0.104 0.096 0.147 0.241
Claude Opus 4.6 0.118 0.096 0.121 0.134 0.379
GLM-5V-Turbo 0.101 0.126 0.107 0.091 0.205
Grok-4.20 0.024 0.000 0.029 0.151 0.102
Kimi-K2.5 0.141 0.183 0.197 0.115 0.171
Qwen3.6-Plus 0.110 0.118 0.078 0.091 0.310
Table 7: Prompt template for oracle slices and notes direct QA. The same template is used for full oracle trajectories.
Component Template
System prompt, web You are an experienced colleague in a web browsing environment that has a customized Magento-based shopping website, a customized Magento-based shopping admin CMS website, as well as a customized forum website based on Reddit/Postmill. Answer based on your memory of the environment. If you do not know the answer, output exactly \boxed{UNKNOWN}. Do not guess. Never attempt to guess an answer if you are not sure. If you believe the question’s construction/premise is wrong, provide an explanation in \boxed{} explaining why the question is flawed.
System prompt, ServiceNow You are an experienced colleague working in a customized ServiceNow environment. Answer based on your memory of the environment. If you do not know the answer, output exactly \boxed{UNKNOWN}. Do not guess. Never attempt to guess an answer if you are not sure. If you believe the question’s construction/premise is wrong, provide an explanation in \boxed{} explaining why the question is flawed.
User prompt # Memory context:
## Procedure and Hint Notes Learned from Previous Tasks in the Environment
For each oracle trajectory: procedure note title, description, and bullet content; hint note title, description, and bullet content.
## Oracle Trajectories and Relevant State Slices from Previous Tasks in the Environment
For each selected trajectory: goal, outcome, start URL, action list, and evidence windows centered at annotated answer states. Each evidence window includes states from radius 1 around the annotated state, with URL, action, accessibility-tree text, and screenshots according to the rendering configuration.
# Question to answer:
{question}
If the question contains an image, the image is appended after the question text.
Table 8: Sandbox layout and instruction for the Codex oracle direct-QA pilot study.
Component Content
Sandbox layout question.json: question text and optional copied question image.
INSTRUCTION.md: task instruction.
answer.json: initialized as {"answer": ""}.
trajectories/{trajectory_id}/trajectory.json: oracle trajectory with id, optional original_goal, optional outcome, and state content.
trajectories/{trajectory_id}/screenshots/: copied trajectory screenshots.
Package instruction You are an experienced colleague working in a customized web environment. Read question.json and inspect every trajectory under trajectories/. This package comes from a public-environment-based setup that has been customized. Do not rely on prior knowledge of the public environment. Work only from the provided trajectories and copied question assets. Use only the question and the provided trajectories. Do not browse anywhere else, do not inspect other question folders, and do not use outside resources. If the question specifies an answer format, follow it exactly. For multiple-choice questions, write only the boxed letter corresponding to your answer, e.g., \boxed{A}, into the answer field. Write your final answer to answer.json using this exact schema: {"answer": "<your final answer>"}.
Codex invocation prompt You are an experienced colleague working in a customized web environment. Read the local files in this directory, especially INSTRUCTION.md and question.json. If question.json refers to a screenshot, view it carefully. Use only local files in this directory. Solve the task and write your final answer to answer.json as valid JSON with a non-empty string field named answer. If answer.json already exists, update only the answer value. Follow the formatting instructions in question.json. For multiple-choice questions, write only the boxed letter corresponding to your answer, e.g., \boxed{A}, into the answer field.
Table 9: Prompt templates used by AgentRunbook-R. Trajectory-specific content is abbreviated.
Prompt Template
Procedure and hint note generation System: You convert one UI task trajectory into two reusable memory notes for a future agent. Assume these notes will later be retrieved for unknown future questions. Preserve the workflow and the highest-value reusable facts from the touched pages. Write procedure_note and hint_note. Each note must contain title, description, and content. Use only evidence grounded in the provided goal, outcome, thoughts, annotated actions, and screenshots. Do not invent unseen fields, filters, modules, or outcomes. If the run failed, describe only the intended or attempted workflow where the evidence supports it. Keep the procedure note focused on the reliable core workflow and use the hint note for durable facts, pitfalls, option sets, confirmation signals, absent functionality, and distinctions between easily confused controls. Return only valid JSON: {"procedure_note":{"title":"...","description":"...","content":"- ..."},"hint_note":{"title":"...","description":"...","content":"- ..."}}. User: Extract two reusable notes from this UI task run. Goal: {goal}. Outcome: {outcome}. Start URL: {start_url}. Each state block is followed by the screenshot for that state. The action line is the action taken from that state, annotated with recoverable object or module details. Then include the ordered state blocks, thoughts, annotated actions, and screenshots.
State-transition event generation System: You convert one UI transition from a longer task trajectory into retrieval-ready event text. You will be given the task goal and outcome, the full annotated action trace, and one target transition defined as pre-state, annotated action, and post-state. Return exactly one JSON object: {"overview":"...","state_transition":"..."}. The overview briefly recaps the task goal and workflow stage. The state-transition field explicitly compares the post-state to the pre-state and describes what changed after the action, such as a new page, revealed panel, form fields, changed values, confirmation signal, blocker, popup, navigation, or lack of visible change. Ground the output only in the provided evidence and preserve exact UI labels when available. User: Generate an event for transition {state_i} -> {state_j}. Goal: {goal}. Outcome: {outcome}. Full action trace: {actions}. Pre-state: {url, thoughts, action, AXTree, screenshot}. Post-state: {url, thoughts, action, AXTree, screenshot}.
Query generation System: You generate structured retrieval queries for an active memory system with three pools: raw state slices, state-transition events, and procedure/hint notes. Return exactly one JSON object: {"raw_state_queries":["..."],"event_query":"...","note_query":"..."}. Maximize retrieval of memory entries that would help answer the question later. Do not answer the question yourself. Use raw-state queries for exact UI surface evidence, such as pages, forms, records, tabs, fields, buttons, dropdowns, options, labels, values, counts, and missing controls. Use the event query only when navigation, before/after change, revealed content, confirmation, blocker, popup, or workflow stage matters. Use the note query for reusable procedures, module paths, disambiguation, absent functionality, pitfalls, and durable hints. Remove formatting instructions and final-answer wrappers. Preserve exact entity names and literal UI labels. Deduplicate raw-state queries and cap them at five. Return JSON only. User: Memory pool summary: {runtime_summary}. Output schema example: {schema_example}. Prompt examples: {few_shot_examples}. Question ID: {question_id}. Question type: {question_type}. Question text: {question}. Question image path: {image_path or <none>}. Original goals attached to this benchmark question: {original_goals}. Return only the JSON object.
Table 10: AgentRunbook-C query-time sandbox layout.
Path Content
question.json Question text, question id, metadata, and optional copied question image path.
INSTRUCTION.md Workflow instruction for using the sandbox as a memory retrieval module.
trajectories/ Symlink to the inserted trajectory haystack.
trajectories/<trajectory_id>/trajectory.json One full trajectory, including goal, start URL, outcome, actions, and ordered states.
trajectories/<trajectory_id>/screenshots/ Screenshots referenced by trajectory states.
trajectories/TRAJECTORY_SUMMARY_CONCISE.md Compact trajectory-level manifest for quick triage.
trajectories/TRAJECTORY_SUMMARY_FULL.md Fuller manifest with detailed thought and action traces for shortlist selection.
scripts/inspect_trajectory.py Helper script for inspecting one trajectory, state, span, or text match.
memory_module_output.json Structured output written by the coding agent.
Table 11: Prompt and workflow instruction for AgentRunbook-C. Codex invocation prompt is the prompt for invoking the codex binary software. The other rows in the table are the content in the INSTRUCTION.md.
Component Template
Codex invocation prompt You are acting as the query-time agent for Coding Agent Memory. Read the local files in this directory, especially INSTRUCTION.md and question.json. The local trajectories/ directory contains the current haystack for this evaluation item, and you must explore trajectories/ before returning your final result. If question.json refers to an image, view it carefully. Write your final result to memory_module_output.json as valid JSON. Use the local inspection helper under scripts/ when you need to inspect one trajectory, one state, one span, or match text within one trajectory quickly.
Task overview in INSTRUCTION.md You are acting as a quick memory retrieval module to provide contexts from agent trajectories collected from a customized web environment for a downstream reader to answer questions specific to that environment. The question is in question.json. Aggregate information from the local trajectories/ directory. Follow the workflow and do not attempt to re-verify or rebuild maps unnecessarily, since the task has latency constraints. Be quick and do not over-explore unless necessary. Work inside the current directory and never explore outside it.
Output requirement Write the final result to memory_module_output.json as valid JSON: {"memory_markdown":"## Support Analysis\n...\n\n## Relevant Procedure and Hint Notes\n...","trajectory_spans":[{"trajectory_id":"...", "start_state_index":0,"end_state_index":0}]}. The support analysis should briefly describe where the evidence can be found. If the evidence contradicts the premise of the question, clearly say that the premise is wrong and include the contradicting evidence. The trajectory spans must use zero-based inclusive indices and preserve order by importance.
Workflow instruction First classify the question before opening trajectories in detail. If the question contains an image, inspect it and align it with the matching surface or state. For direct lookup questions, find an exact state showing the requested field, value, button, or page. For comparison questions, find one supporting state per side when needed. For procedure questions, stay within the same workflow family unless the question explicitly asks for a shared pattern across workflows. Start from TRAJECTORY_SUMMARY_FULL.md and shortlist only a few likely trajectories using the goal, start URL, action sequence, and final reward. Prefer the helper script for exact verification: python scripts/inspect_trajectory.py <trajectory_id>, --state <i>, --span <i:j>, or --match "<pattern>". Use the helper on shortlisted trajectories rather than performing broad raw-file search. Keep the final evidence package small, usually no more than three states per span, and use at most 20 states in total.
Final rules Move fast and prefer targeted exploration. Put the most important evidence first. Avoid redundant trajectories when multiple trajectories support the same fact. Reject nearby but non-exact matches. Do not copy screenshots or large AXTree blocks into the JSON output. You may write scratch files in the current directory if needed.
Table 12 : Successful AgentRunbook-C examples with the selected question markdown, evidence span content, and corresponding screenshot.
Question type Question markdown Selected span content Screenshot
Static environment QID: 98b62f3d. Question. I am using our reddit-based custom forum website. For the create submission form, what are the names of the mandatory fields? Mark your final answer as a comma-separated list of short phrases in \boxed{}. Gold / reader answer. Title, Forum. Support analysis. The clearest evidence is trajectory 4ba5e9cb, state 2, on the Create submission form for /submit/pittsburgh. That state marks Title and Forum as required, while Body is optional and the URL/Image controls are submission-type selectors. Trajectory state span. 4ba5e9cb: states 2–2. State 2 AXTree excerpt. [136] LabelText: Title *; [138] textbox Title This field is required., required; [141] LabelText: Body; [143] textbox Body; [148] checkbox Formatting help +; [277] LabelText: Forum *; [384] combobox value pittsburgh. [Uncaptioned image] Trajectory 4ba5e9cb, state 2.
Dynamic environment QID: 609acb91. Question. I am using our magento-based custom shopping website. I am now browsing the item list that appears after I click a specific item category from the home page. If I then narrow the display scope by selecting a specific price range in the left column, two new links will appear after the selected range. What are the names of the links? Your final answer should be a comma-separated list of two phrases wrapped in \boxed{}. Gold / reader answer. Remove This Item, Clear All. Support analysis. The supporting evidence is trajectory dddd8aa2, state 5. That state shows the Men >> Shoes category page after the Price: $0.00–$29.99 filter is applied, and immediately after the selected range the sidebar lists the two links Remove This Item and Clear All. Trajectory state span. dddd8aa2: states 5–5. The action sequence opens Men >> Shoes, selects Price, and loads the filtered page with price=0-30. State 5 AXTree excerpt. [1922] strong: Shop By; [1925] heading: Now Shopping by; [1927] listitem: Price: $0.00–$29.99; [1930] link: Remove This Item, clickable, visible; [1933] link: Clear All, clickable, visible. [Uncaptioned image] Trajectory dddd8aa2, state 5.
Table 13 : Successful AgentRunbook-R examples where different memory pools provide the answer-bearing evidence. The controller queries and retrieved items are copied from the corresponding runs.
Pool Question Controller query Retrieved evidence from that pool Figure
Procedure and hint notes Magento storefront order-history question: the user is already on My Orders; the table says “Items 1 to 10 of 37 total”; which pagination label should be clicked first to reach the oldest orders most directly? Choices: A. 2, B. 3, C. 4, D. Next, E. Last. Correct answer: C. Magento My Orders page pagination and date sorting behavior for finding oldest purchase Procedure note, rank 1 (score 0.7465, trajectory 19022110) is titled Find Earliest Purchase Date in My Orders. It states that on My Orders, the reader should note the pagination text, compute the final page from the total count, click the final page number, and verify the final range, e.g., “Items 31 to 37 of 37 total.” The paired hint note also records that oldest orders are on the last page. With 37 orders and 10 per page, the first direct click is page 4. [Uncaptioned image] Trajectory 19022110: final order-history page.
State-transition events Postmill forum dynamic question: after replying to a nested comment, a blue banner appears above the reply; what does the link in that banner say? Correct answer: View all comments. submit comment reply to nested thread and observe blue banner overlay appearance Event result 2 (similarity 0.5923, trajectory 2e8f6477) retrieves the transition from state 9 to state 10 after the nested reply was posted. The event is attached to the single-comment-thread view for the AskReddit post and includes the post-state screenshot where the banner text is visible. The banner reads “Viewing a single comment thread.” and its link reads “View all comments.” [Uncaptioned image] Trajectory 2e8f6477: reply result banner.
Raw state slices ServiceNow form-comparison question: between Create Change Request and Incident, what additional top-right button appears on the incident form but not the change-request form? Correct answer: Resolve. ServiceNow Incident form top right button area visible controls; ServiceNow Change Request form top right button area visible controls Raw state result 1 (similarity 0.7168, trajectory 454485ca, center state 7) retrieves the incident creation form; the top-right controls include Submit and Resolve. Raw state result 4 (similarity 0.7087, trajectory afa62eac, center state 6) retrieves the change-request creation form; its top-right controls show Submit without Resolve. The contrast isolates the extra incident-only button. [Uncaptioned image] Incident: Submit, Resolve.
[Uncaptioned image]
Change Request: Submit only.