← 返回阅读笔记 / Back to reading note

STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments 原表

Y26-124 原文表格

抽取日期:2026-09-07。来自原文。保留原表行列,未统一实验配置;数值解释见中文分析。

Table 1: Main results on AndroidWorld-M . We report task success accuracy under pass@1 and pass@3.
Model pass@1 pass@3
L1 L2 L3 Avg. L1 L2 L3 Avg.
General-purpose Multimodal Model + Active Memory Instruction (AMI)
Qwen3-VL-8B-Instruct (Bai et al., 2025) 15.4 7.7 0.0 7.7 23.1 15.4 7.7 15.4
Qwen3-VL-8B-Think (Bai et al., 2025) 23.1 15.4 0.0 12.8 30.8 23.1 7.7 20.5
GPT-5.4 (OpenAI, 2026) 23.1 15.4 7.7 15.4 23.1 15.4 7.7 15.4
Gemini-3.1-Pro (Deepmind, 2026) 30.8 15.4 23.1 23.1 38.4 23.1 23.1 28.2
Claude-Sonnet-4.6 (Anthropic, 2026) 46.2 46.2 38.5 43.6 46.2 53.8 46.2 48.7
GUI-specialized Model + History-Retention Protocol (HRP)
CogAgent-9B (Hong et al., 2024) 0.0 0.0 0.0 0.0 7.7 0.0 0.0 2.6
UI-Tars-1.5-7B (Qin et al., 2025) 7.7 7.7 7.7 7.7 7.7 15.4 15.4 12.8
UI-Venus-1.5-8B (Team et al., 2026) 7.7 0.0 0.0 2.6 15.4 7.7 7.7 10.3
MAI-UI-8B (Zhou et al., 2025) 15.4 7.7 7.7 10.3 15.4 15.4 7.7 12.8
GUI-Owl-1.5-8B-Think (Xu et al., 2026) 15.4 7.7 7.7 10.3 23.1 15.4 7.7 15.4
Ours (Training from GUI-Owl-1.5-8B-Think)
Stamp-GUI 46.2 30.8 15.4 30.8 53.8 38.4 23.1 38.4
Table 2: Main results on MemGUI-Bench . We report task success accuracy under pass@1 and pass@3.
Model pass@1 pass@3
L1 L2 L3 Avg. L1 L2 L3 Avg.
General-purpose Multimodal Model + Active Memory Instruction (AMI)
Qwen3-VL-8B-Instruct (Bai et al., 2025) 11.1 0.0 7.4 6.2 25.9 7.4 18.5 17.3
Qwen3-VL-8B-Think (Bai et al., 2025) 18.5 7.4 3.7 9.9 40.7 14.8 14.8 23.4
GPT-5.4 (OpenAI, 2026) 37.0 30.8 33.3 33.7 70.4 59.3 74.1 67.9
Gemini-3.1-Pro (Deepmind, 2026) 48.1 37.0 22.2 35.8 74.1 51.9 44.4 56.8
Claude-Sonnet-4.6 (Anthropic, 2026) 66.7 70.4 51.9 63.0 77.8 77.8 70.4 75.3
GUI-specialized Model + History-Retention Protocol (HRP)
CogAgent-9B (Hong et al., 2024) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
UI-Tars-1.5-7B (Qin et al., 2025) 7.4 7.4 0.0 4.9 14.8 14.8 7.4 12.3
UI-Venus-1.5-8B (Team et al., 2026) 3.7 7.4 14.8 8.6 18.5 11.1 25.9 18.5
MAI-UI-8B (Zhou et al., 2025) 22.2 14.8 18.5 18.5 29.6 22.2 22.2 24.7
GUI-Owl-1.5-8B-Think (Xu et al., 2026) 22.2 18.5 18.5 19.7 25.9 22.2 25.9 24.7
Ours (Training from GUI-Owl-1.5-8B-Think)
Stamp-GUI 25.9 22.2 22.2 23.4 37.0 33.3 40.7 37.0
Table 3: Main results on Memory-World . We report memory accuracy (M-Acc) and task accuracy (T-Acc) under pass@1 and pass@3.
Model pass@1   pass@3  
T-Acc M-Acc T-Acc M-Acc
General-purpose Multimodal Model + Active Memory Instruction (AMI)  
Qwen3-VL-8B-Instruct (Bai et al., 2025) 16.5 46.6 19.7 50.4
Qwen3-VL-8B-Think (Bai et al., 2025) 36.2 60.6 41.7 64.6
GPT-5.4 (OpenAI, 2026) 52.0 85.4 56.7 89.8
Gemini-3.1-Pro (Deepmind, 2026) 49.6 79.6 55.2 87.4
Claude-Sonnet-4.6 (Anthropic, 2026) 59.1 95.2 67.7 98.5
GUI-specialized Model + History-Retention Protocol (HRP)  
CogAgent-9B (Hong et al., 2024) 19.7 24.4 22.1 26.8
UI-Tars-1.5-7B (Qin et al., 2025) 20.5 33.9 26.8 37.4
UI-Venus-1.5-8B (Team et al., 2026) 18.9 21.2 22.0 24.4
MAI-UI-8B (Zhou et al., 2025) 40.2 51.2 45.7 56.7
GUI-Owl-1.5-8B-Think (Xu et al., 2026) 44.9 55.9 52.0 59.8
Ours (Training from GUI-Owl-1.5-8B-Think)  
Stamp-GUI 75.3 77.9 82.6 84.2
Table 4: Ablation results. We incrementally add each component in STAMP on training baseline. For AndroidWorld-M and MemGUI-Bench, we report the average task accuracy over L1–L3 under pass@1 and pass@3.
Method AndroidWorld-M MemGUI-Bench Memory-World (pass@1) Memory-World (pass@3)
pass@1 pass@3 pass@1 pass@3 T-Acc M-Acc T-Acc M-Acc
Baseline 10.3 15.4 19.7 24.7 44.9 55.9 52.0 59.8
+ Virtual Env 20.8 31.1 22.2 30.8 57.5 68.3 63.1 71.9
+ Step Balance 28.2 38.4 23.4 32.1 69.4 74.8 73.2 79.4
+ A/M Critic 25.7 35.9 21.0 39.5 74.3 77.4 81.0 83.6
+ Online RL 30.8 38.4 23.4 37.0 75.3 77.9 82.6 84.2
Table 5: Effect of step balance. We report the average accuracy over L1–L3 under pass@1/pass@3.
Method AW-M MemGUI Memory-World
Avg. Avg. T-Acc M-Acc
Vanilla 20.8/31.1 22.2/30.8 57.5/63.1 68.3/71.9
Up to 1:1 25.7/41.0 22.2/34.6 71.7/78.4 73.3/81.1
Up to 3:1 30.8/38.4 23.4/37.0 75.3/82.6 77.9/84.2
Table 6: Dynamic testing results. SR denotes success rate, Avg. Step denotes the average steps, and Correct Step denotes the proportion of correct steps.
Model SR Avg. Step Correct Step
GPT-5.4 69.8 15.4 83.6
Gemini-3.1-Pro 74.5 13.2 87.2
Claude-Sonnet-4.6 77.4 12.7 89.5
Table 7: User prompt template used by Stamp-GUI.
User Prompt
Please generate the next move according to the UI screenshot, instruction and previous actions.
Instruction: {goal}
Previous actions:
{history_string}
Table 8: Main optimization hyperparameters used in online RL.
  Item   Value
  Optimizer learning rate   2×1072\times 10^{-7}
  Train batch size   32
  PPO mini-batch size   32
  Micro-batch size per GPU   1
  Policy loss mode   PPO
  Clip ratio (low / high)   0.05 / 0.05
  KL loss   Enabled
  KL coefficient   0.05
  KL type   Low-variance KL
  Entropy coefficient   5×1045\times 10^{-4}
  Gradient checkpointing   Enabled
  Critic warmup   0
Table 9: Example of a platform seed used for virtual environment generation.
Field Value
Type social
App name StarWave
Brand slogan Discover every trend in everyday life
Style [Feed, Search, Profile, Messages, Me]
Primary color #FF5C8A
Secondary color #7A3CFF
Background color #F7F8FA
Card style Rounded cards
Icon style Soft skeuomorphic
Common entities [creator, topic, hot post, followers, likes, shares]
Detail entry points [post detail, topic detail, pinned post on creator profile]
Distractions [Trending Picks, You May Like, Upgrade to Premium, Find Contacts, Live Stream]
Text tone Young, social, energetic
Table 10: Example of a task seed used for virtual task instantiation.
Field Value
ID date_compare_latest
Description Read dates from multiple detail pages, find the latest one, and submit the corresponding name
Goal Compare date fields and output the name associated with the latest date
Required steps 4
Memory load Medium
Output format Output only the latest item’s name
Answer type single_label
UI pattern [Detail A, Detail B, Detail C, Submit name]
Table 11: Prompt used by the dynamic testing worker.
Dynamic Testing Worker Prompt
You are an expert in mobile phone operations. I will provide you with a mobile screenshot and an operation task, and you need to output the exact action to perform.
Mobile screenshot: shown in the image below
Operation task: {next_planning}
Your action space is as follows. You must choose one of the following actions and output it in the required format:
- click (x, y): click the position whose coordinates are (x, y), where both x and y are relative coordinates in the range 0-1000
- type (x, y, text): first click the position whose coordinates are (x, y), then input text, where text is the content to type
- scroll: scroll the screen downward a bit to view more content that is not currently visible below
Your output format should follow the examples below:
click (500, 500)
type (100, 100, hello)
scroll
Note: Please do not attempt to manipulate elements at the very top of the page (the browser’s fixed tag bar, y<=120). All your actions should be concentrated in the webpage content area.
Do not output anything other than the formats above. You may output only one action at a time. Note that type (x, y, text) represents two actions: first clicking (x, y), then typing text.
Table 12: Task-level details of AndroidWorld-M. Each difficulty level contains the same 13 task types, while L1 and L2 use more explicit task instructions and L1 additionally introduces controlled simplifications.
Task ID Type L1 & L2 Instruction L3 Instruction L1 Simplification Note
BrowserMultiply Native Open the file task.html in Downloads in the file manager; when prompted open it with Chrome. Then click the button that says “click me” 5 times, remember the 5 numbers displayed (may be same), and then make sure you see the submit page after clicking 5 times. Finally, submit the product of the 5 numbers in the form. Open the file task.html in Downloads in the file manager; when prompted open it with Chrome. Then click the button 5 times, remember the numbers displayed, and enter their product in the form. Replace the 5 numbers with fixed values 1, 2, 3, 4, 5.
ExpenseAddSingleFromGallery Ext. Go to Simple Gallery Pro and search for expenses.jpg in DCIM to remember the expenses on expenses.jpg (first go to the DCIM folder, then use the search bar to search for expenses.jpg, finally make sure you are looking at expenses.jpg and remember the three expenses). Then add the remembered three expenses to {_APP_NAME} (make sure each expense has correct Name, Amount, category, and Note). Add the expenses from expenses.jpg in Simple Gallery Pro to {_APP_NAME}. Reduce Gallery noise images from 10 to 3.
ExpenseAddMultipleFromGallery Native Go to Simple Gallery Pro and search for expenses.jpg in DCIM to remember the expenses on expenses.jpg (first go to the DCIM folder, then use the search bar to search for expenses.jpg, finally make sure you are looking at expenses.jpg and remember the three expenses). Then add the remembered three expenses to {_APP_NAME} (make sure each expense has correct Name, Amount, category, and Note). Add the expenses from expenses.jpg in Simple Gallery Pro to {_APP_NAME}. Reduce target expenses from 3 to 2; Gallery noise from 10 to 3.
ExpenseAddSingleFromMarkor Ext. Go to Markor and find my_expenses.txt to remember the one reimbursable transaction (whose note includes “Reimbursable.”; if you do not find the reimbursable transaction, please scroll the page). Then add the remembered reimbursable transaction to {_APP_NAME} (make sure each expense has correct Name, Amount, category, and Note). Go through the transactions in my_expenses.txt in Markor. Log the reimbursable transactions in the {_APP_NAME}. Reduce Markor noise entries from 100 to 1.
ExpenseAddMultipleFromMarkor Native Go to Markor and find my_expenses.txt to remember the one reimbursable transaction (whose note includes “Reimbursable.”; if you do not find the reimbursable transaction, please scroll the page). Then add the remembered reimbursable transaction to {_APP_NAME} (make sure each expense has correct Name, Amount, category, and Note). Go through the transactions in my_expenses.txt in Markor. Log the reimbursable transactions in the {_APP_NAME}. Reduce target expenses from 3 to 2; Markor noise from 100 to 1.
MarkorTranscribeReceipt Native Go to Simple Gallery app to view the receipt.png, and remember the receipt on it. Then go to Markor app to create a new file called receipt.md and enter transactions in csv format including the header “Date, Item, Amount”. Create a file in Markor, called receipt.md with the transactions from the receipt.png. Use Simple Gallery to view the receipt. Please enter transactions in csv format including the header “Date, Item, Amount”. Increase max step limit from 18 to 25.
MarkorTranscribeVideo Native Transcribe the contents of video {video_name} by watching it in VLC player (located in Download) and writing the sequence of strings shown on each frame to the text file {file_name} in Markor as a comma separated list. Transcribe the contents of video {video_name} by watching it in VLC player (located in Download) and writing the sequence of strings shown on each frame to the text file {file_name} in Markor as a comma separated list. For example, if the first frame shows the text “edna” and the second frame shows the text “pineapple”, then the text file should contain only the following text: “edna, pineapple”. Increase max step limit from 20 to 40.
RecipeAddSingleRecipesFromImage Ext. Go to Simple Gallery Pro and open recipes.jpg in DCIM to remember all the information of the three recipes (first go to the DCIM folder, then just click the recipes.jpg on the top left once to open it, finally make sure you are looking at recipes.jpg and remember the three recipes). Then add the remembered three recipes to Broccoli recipe app. Please make sure the name, description, servings, preparation time, ingredients, and directions are entered correctly. Add the recipes from recipes.jpg in Simple Gallery Pro to the Broccoli recipe app. Reduce Gallery noise images from 10 to 1.
RecipeAddMultipleRecipesFromImage Native Go to Simple Gallery Pro and open recipes.jpg in DCIM to remember all the information of the three recipes (first go to the DCIM folder, then just click the recipes.jpg on the top left once to open it, finally make sure you are looking at recipes.jpg and remember the three recipes). Then add the remembered three recipes to Broccoli recipe app. Please make sure the name, description, servings, preparation time, ingredients, and directions are entered correctly. Add the recipes from recipes.jpg in Simple Gallery Pro to the Broccoli recipe app. Reduce target recipes from 3 to 2; Gallery noise from 10 to 1.
RecipeAddSingleRecipesFromMarkor Ext. Go to Markor and find recipes.txt to remember all the information (name, description, servings, preparation time, ingredients, and directions) of the recipe. Then add the remembered recipe to Broccoli recipe app. Please make sure the name, description, servings, preparation time, ingredients, and directions are entered correctly. Add the recipes from recipes.txt in Markor to the Broccoli recipe app. Reduce Markor noise entries from 10 to 1.
RecipeAddMultipleRecipesFromMarkor Native Go to Markor and find recipes.txt to remember all the information (name, description, servings, preparation time, ingredients, and directions) of the three recipes. Then add the remembered three recipes to Broccoli recipe app. Please make sure the name, description, servings, preparation time, ingredients, and directions are entered correctly. Add the recipes from recipes.txt in Markor to the Broccoli recipe app. Reduce target recipes from 3 to 2; Markor noise from 10 to 1.
RecipeAddSingleRecipesFromMarkor2 Ext. Go to Markor and find recipes.txt to remember all the information (name, description, servings, ingredients, and directions) of the recipe whose preparation time is {prep_time}. Then add the remembered recipe to Broccoli recipe app. Please make sure the name, description, servings, ingredients, and directions are entered correctly. Please make sure that the preparation time is {prep_time} for the recipe. Add the recipes from recipes.txt in Markor that take {prep_time} to prepare into the Broccoli recipe app. Reduce Markor noise entries from 40 to 1.
RecipeAddMultipleRecipesFromMarkor2 Native Go to Markor and find recipes.txt to remember all the information (name, description, servings, ingredients, and directions) of the recipe whose preparation time is {prep_time}. Then add the remembered recipe to Broccoli recipe app. Please make sure the name, description, servings, ingredients, and directions are entered correctly. Please make sure that the preparation time is {prep_time} for the recipe. Add the recipes from recipes.txt in Markor that take {prep_time} to prepare into the Broccoli recipe app. Reduce target recipes from 3 to 2; Markor noise from 40 to 1.
Table 13: Task descriptions of MemGUI-Bench under the three-level protocol (Part I).
Task ID L1 Task L2 Task L3 Task
011-SetMeetingAlarmByWorldClock Open the Clock app and go to World Clock. Then add ‘Beijing, China’ and ‘New York, USA’ to World Clock. Next, compare the time differences between the two locations and choose any Beijing time that is between 8 AM and 10 PM in both Beijing and New York. Finally, add this Beijing time to your alarm for tomorrow. Open the Clock app and go to World Clock. Then add ‘Beijing, China’ and ‘New York, USA’ to World Clock. Next, compare the time differences between the two locations and choose any Beijing time that is between 8 AM and 10 PM in both Beijing and New York. Finally, directly answer this Beijing time. Open the Clock app and add ‘Beijing, China’ and ‘New-York, USA’ to the world clock. By comparing their current times, find a suitable meeting time for tomorrow that falls between 8 AM and 10 PM in both cities. Then, set an alarm for this meeting, using the local time in Beijing as the reference.
012-SetMeetingAlarmByWorldClock Open the Clock app and go to World Clock. Then add ‘Beijing, China’ and ‘Buenos Aires, Argentina’ to World Clock. Next, compare the time differences between the two locations and choose any Beijing time that is between 9 AM and 11 PM in both Beijing and Buenos Aires. Finally, add this Beijing time to your alarm for tomorrow. Open the Clock app and go to World Clock. Then add ‘Beijing, China’ and ‘Buenos Aires, Argentina’ to World Clock. Next, compare the time differences between the two locations and choose any Beijing time that is between 9 AM and 11 PM in both Beijing and Buenos Aires. Finally, directly answer this Beijing time. Open the Clock app and add ‘Beijing, China’ and ‘Buenos Aires, Argentina’ to the world clock. By comparing their current times, find a suitable meeting time for tomorrow that falls between 9 AM and 11 PM in both cities. Then, set an alarm for this meeting, using the local time in Beijing as the reference.
025-FindAndCompareWikiStats Open the Wikipedia app. First, search for ‘English Wikipedia’, scrolling until you find the current number of articles. Remember this number and back to homepage. Then search for ‘German Wikipedia’, scrolling until you find the current number of articles. Remember this number and back to homepage. Then compare the two numbers to see which has more articles. Finally, search again for it (‘English Wikipedia’ or ‘German Wikipedia’) with more articles and go to its details page. Open the Wikipedia app. First, search for ‘English Wikipedia’, scrolling until you find the current number of articles. Remember this number and back to homepage. Then search for ‘German Wikipedia’, scrolling until you find the current number of articles. Remember this number and back to homepage. Then compare the two numbers to see which has more articles. Finally, directly answer which Wikipedia (‘English Wikipedia’ or ‘German Wikipedia’) has more articles. Open the Wikipedia app, search for ‘English Wikipedia’ and find the current number of articles it contains. Remember this number. Then, search for ‘German Wikipedia’ and find its current number of articles. Go to and stay on the page of the Wikipedia edition that has more articles.
026-FindAndCompareWikiStats Open the Wikipedia app. First, search for ‘Beijing’, scrolling until you find the population. Remember this number and back to homepage. Then search for ‘Shanghai’, scrolling until you find the population. Remember this number and back to homepage. Finally, search again for it (‘Beijing’ or ‘Shanghai’) with the larger population and go to its page. Open the Wikipedia app. First, search for ‘Beijing’, scrolling until you find the population. Remember this number and back to homepage. Then search for ‘Shanghai’, scrolling until you find the population. Remember this number and back to homepage. Finally, directly answer which city (‘Beijing’ or ‘Shanghai’) has larger population. Open the Wikipedia app, first search for ‘Beijing’ and remember its population. Next, search for ‘Shanghai’ and remember its population. Compare the two, and then go to and remain on the Wikipedia page for the city with the larger population.
027-NavigateAndCompareItems Open the Wish app. Go to the ‘Jewelry & watches’ category. First, go to the ‘Fashion jewelry’ subcategory, and then select the cheapest item from the first 6 results. Remember the number of reviews for this item. Next, go to the ‘Watches’ subcategory, and again select the cheapest item from the first 6 results. Remember the number of reviews for this item. Compare the reviews of the two items you remembered. Finally, go back to the product page of the item with more reviews. Open the Wish app. Go to the ‘Jewelry & watches’ category. First, go to the ‘Fashion jewelry’ subcategory, and then select the first item from the first 6 results. Remember the number of reviews for this item. Next, go to the ‘Watches’ subcategory, and again select the first item from the first 6 results. Remember the number of reviews for this item. Compare the reviews of the two items you remembered. Finally, directly answer which item has more reviews. In the Wish app, identify two items: the cheapest in ‘Jewelry & watches’ \rightarrow ‘Fashion jewelry’ (from top 6), and the cheapest in the ‘Watches’ sub-category (from top 6). Remember the number of reviews for both. Navigate to the page of whichever of these two items has more reviews.
028-NavigateAndCompareItems Open the Wish app. Go to the ‘Office & tech’ category. Then, go to the ‘Parts & storage’ subcategory and select the product with the most reviews from the first six results. Remember the price of this product. Next, go to the ‘Hardware’ subcategory and select the product with the most reviews from the first six results. Remember the price of this product. Compare the prices of the two products you remembered. Finally, go back to the product page of the cheaper one. Open the Wish app. Go to the ‘Office & tech’ category. Then, go to the ‘Parts & storage’ subcategory and select the product with the most reviews from the first six results. Remember the price of this product. Next, go to the ‘Hardware’ subcategory and select the product with the most reviews from the first six results. Remember the price of this product. Compare the prices of the two products you remembered. Finally, directly answer which product is cheaper. In the Wish app, identify two items: the one with the most reviews in ‘Office & tech’ \rightarrow ‘Parts & storage’ (from top 6), and the one with the most reviews in the ‘Hardware’ sub-category (from top 6). Remember the price for both. Navigate to the page of whichever of these two items is cheaper.
035-CheckProductCompatibility Open Amazon app, search for ‘Anker 737 Power Bank’. Remember its price, capacity (in mAh), and all output port types (make sure all information is found; otherwise, continue searching). Then, back to homepage and search for ‘MacBook Air M2’ and remember its charging port. Next, back to homepage and search for ‘iPhone 15 Pro’ and remember its charging port. Finally open Joplin app, note all the remembered data and answer if the ‘Anker 737 Power Bank’ can simultaneously charge both ‘MacBook Air M2’ and ‘iPhone 15 Pro’. Open Amazon app, search for ‘Anker 737 Power Bank’. Remember its output port types. Then, back to homepage and search for ‘MacBook Air M2’ and remember its charging port. Next, back to homepage and search for ‘iPhone 15 Pro’ and remember its charging port. Finally, directly answer if the ‘Anker 737 Power Bank’ can simultaneously charge both ‘MacBook Air M2’ and ‘iPhone 15 Pro’. Open Amazon app, search for ‘Anker 737 Power Bank’. Find its price, capacity (in mAh), and all output port types. Then, search for ‘MacBook Air M2’ and find its required charging port. Next, search for ‘iPhone 15 Pro’ and find its charging port. In Joplin app, note all the collected data and answer if the power bank can simultaneously charge both devices.
036-CheckProductCompatibility Open Amazon app, search for ‘Samsung T7 Shield SSD’. Remember its USB port specifications, storage capacity, and read/write speeds (make sure all information is found; otherwise, continue searching). Then, back to homepage and search for ‘PlayStation 5’ and remember its USB port specifications. Next, back to homepage and search for ‘Xbox Series X’ and remember its USB port specifications. Finally open Joplin app, note all the remembered data and answer if the ‘Samsung T7 Shield SSD’ is fully compatible with both ‘PlayStation 5’ and ‘Xbox Series X’ consoles’ USB standards for external storage. Open Amazon app, search for ‘Samsung T7 Shield SSD’. Remember its USB port specifications. Then, back to homepage and search for ‘PlayStation 5’ and remember its USB port specifications. Next, back to homepage and search for ‘Xbox Series X’ and remember its USB port specifications. Finally, directly answer if the ‘Samsung T7 Shield SSD’ is fully compatible with both ‘PlayStation 5’ and ‘Xbox Series X’ consoles’ USB standards for external storage. Open Amazon app, search for ‘Samsung T7 Shield SSD’. Find its price, storage capacity, and read/write speeds. Then search for ‘PlayStation 5’ and find its USB port specifications. Next, search for ‘Xbox Series X’ and find its USB port specifications. In Joplin app, note all collected data and answer if the SSD is fully compatible with both consoles’ USB standards for external storage.
041-CheckRamCompatibilityAndPrice Open the Amazon app, search for “32GB DDR5 RAM”, filter for the “Corsair” brand. Remember the price and clock speed of the first item in the search results. Then, open the Bing app and directly search for “ASUS ROG Strix Z790-E motherboard maximum supported memory speed” by typing this text. Finally, confirm if the clock speed of that Corsair 32GB DDR5 RAM is less than or equal to the ASUS ROG Strix Z790-E motherboard’s maximum supported speed. Directly answer with its price if it is compatible, or “Not compatible” if it isn’t. Open the Amazon app, search for “32GB DDR5 RAM”, filter for the “Corsair” brand. Remember the clock speed of the first item in the search results. Then, open the Bing app and directly search for “ASUS ROG Strix Z790-E motherboard maximum supported memory speed” by typing this text. Finally, confirm if the clock speed of that Corsair 32GB DDR5 RAM is less than or equal to the ASUS ROG Strix Z790-E motherboard’s maximum supported speed. Directly answer with its price if it is compatible, or “Not compatible” if it isn’t. Open the Amazon app, search for “32GB DDR5 RAM”, filter for the “Corsair” brand, and remember the price and clock speed of the first item in the search results. Then, open the Bing app and search for “ASUS ROG Strix Z790-E motherboard maximum supported memory speed”. Finally, confirm if the clock speed of that Corsair RAM is less than or equal to the motherboard’s maximum supported speed. Directly answer with its price if it is compatible, or “Not compatible” if it isn’t.
042-CheckSsdCompatibilityAndPrice Open the Amazon app, search for “1TB External SSD”, filter for the “Samsung” brand. Remember the clock speed of the first item in the search results. Then, open the Bing app and directly search for “PlayStation 5 external SSD speed requirement” by typing this text. Finally, confirm if the Samsung 1TB External SSD’s speed meets or exceeds the PlayStation 5’s requirement. Directly answer with its price if it is compatible, or “Not compatible” if it isn’t. Open the Amazon app, search for “1TB External SSD”, filter for the “Samsung” brand. Remember the clock speed of the first item in the search results. Then, open the Bing app and directly search for “PlayStation 5 external SSD speed requirement” by typing this text. Finally, confirm if the Samsung 1TB External SSD’s speed meets or exceeds the PlayStation 5’s requirement. Directly answer with its price if it is compatible, or “Not compatible” if it isn’t. Open the Amazon app, search for “1TB External SSD”, filter for the “Samsung” brand, and remember the price and the read/write speed of the first item in the search results. Then, open the Bing app and search for “PlayStation 5 external SSD speed requirement”. Finally, confirm if the SSD’s speed meets or exceeds the PS5’s requirement. Directly answer with its price if it is compatible, or “Not compatible” if it isn’t.
063-SearchImageAndDescribe Open the bing app, switch to “image”, and then perform an image search for ‘Global Smartphone Shipments Market Share 2021’. From the image results, find the image with the chart that specifically displays the data for Q3 2021. Click this image to open it, then identify the top three brands from this Q3 chart and remember their names and their exact market share percentages. Finally, open the joplin app, create a new note titled ‘Smartphone Market Share 2021 Q3’, and write the top three brands with their corresponding percentages. Open the bing app, switch to “image”, and then perform an image search for ‘Global Smartphone Shipments Market Share 2021’. From the image results, find the image with the chart that specifically displays the data for Q3 2021. Click this image to open it, then identify the top three brands from this Q3 chart and remember their names and their exact market share percentages. Finally, directly answer the top three brands with their corresponding percentages. Open the bing app and perform an image search for ‘Global Smartphone Shipments Market Share 2021’. From the image results, locate and carefully analyze the chart that specifically displays the data for Q3 2021. Identify the top three brands from this Q3 chart, and remember their names and their exact market share percentages. Finally, open the joplin app, create a new note titled ‘Smartphone Market Share 2021 Q3’, and list the top three brands with their corresponding percentages.
064-SearchImageAndCreateNote Open the bing app, switch to “image”, and then perform an image search for ‘Global Vehicle Sales Trend by region November 2023’. From the image results, find the first image with clear chart. Click this image to open it, then identify the top three regions with the highest sales growth or volume and remember names of these regions and their corresponding data values (e.g., sales numbers or percentage growth). Finally, open the joplin app, create a new note titled ‘Vehicle Sales Trend November 2023’, and write the top three regions with their data. Open the bing app, switch to “image”, and then perform an image search for ‘Global Vehicle Sales Trend by region November 2023’. From the image results, find the first image with clear chart. Click this image to open it, then identify the top three regions with the highest sales growth or volume and remember names of these regions and their corresponding data values (e.g., sales numbers or percentage growth). Finally, directly answer the top three regions with their data. Open the bing app and perform an image search for ‘Global Vehicle Sales Trend by region November 2023’. Carefully analyze the first clear chart that appears in the search results. From this chart, identify the top three regions with the highest sales growth or volume, and remember the names of these regions and their corresponding data values (e.g., sales numbers or percentage growth). Finally, open the joplin app, create a new note titled ‘Vehicle Sales Trend November 2023’, and list the top three regions with their data.
Table 14: Task descriptions of MemGUI-Bench under the three-level protocol (Part II, continued from Table 13 ).
Table 13 continued.
067-PlanAndMessageRoute Open the Citymapper app, first search for the journey from the White House to the Lincoln Memorial, and remember the fastest one among Public Transit, Walking, and Taxi. Then search for the journey from the Lincoln Memorial to the National Air and Space Museum, and remember the fastest one among Public Transit, Walking, and Taxi. Determine the fastest possible total travel time by combining the modes for two leg. Finally, send a message to +8613811118888 stating this fastest route combination and the total time. Open the Citymapper app, first search for the journey from the White House to the Lincoln Memorial, and remember the fastest one among Public Transit, Walking, and Taxi. Then search for the journey from the Lincoln Memorial to the National Air and Space Museum, and remember the fastest one among Public Transit, Walking, and Taxi. Determine the fastest possible total travel time by combining the modes for two leg. Finally, directly answer the fastest route combination and the total time. In the Citymapper app, find the travel times for a two-leg journey in Washington D.C.: 1) The White House to the Lincoln Memorial, and 2) the Lincoln Memorial to the National Air and Space Museum. For each leg, find the times for Public Transit, Walking, and Taxi. Determine the fastest possible total travel time by combining the modes for each leg. Send a message to +8613811118888 stating this fastest route combination and the total time.
068-PlanAndMessageRoute Open the Citymapper app, first search for the journey from the Statue of Liberty to the Empire State Building, and remember the fastest one among Public Transit, Walking, and Taxi. Then search for the journey from the Empire State Building to the Metropolitan Museum of Art, and remember the fastest one among Public Transit, Walking, and Taxi. Determine the fastest possible total travel time by combining the modes for two leg. Finally, send a message to +8613822229999 stating this fastest route combination and the total time. Open the Citymapper app, first search for the journey from the Statue of Liberty to the Empire State Building, and remember the fastest one among Public Transit, Walking, and Taxi. Then search for the journey from the Empire State Building to the Metropolitan Museum of Art, and remember the fastest one among Public Transit, Walking, and Taxi. Determine the fastest possible total travel time by combining the modes for two leg. Finally, directly answer the fastest route combination and the total time. In the Citymapper app, find the travel times for a two-leg journey in New York City: 1) Statue of Liberty to the Empire State Building, and 2) the Empire State Building to The Metropolitan Museum of Art. For each leg, find the times for Public Transit, Walking, and Taxi. Determine the fastest possible total travel time by combining the modes for each leg. Send a message to +8613822229999 stating this fastest route combination and the total time.
077-CheckAndNotePermissions Open the Setting app. Search for ‘Special app access’ and open it. First, view the apps with ‘Wi-Fi control’ permission and remember the list of app names. Next, back to previous page and continue to view the apps allowed ‘Picture-in-picture’ access and remember that list of names. Finally, open the joplin app, create a note titled ‘App Access Permissions’, and write the names you remembered under two headings: ‘Wi-Fi Control’ and ‘Picture-in-picture’. Open the Setting app. Search for ‘Special app access’ and open it. First, view the apps with ‘Wi-Fi control’ permission and remember the list of app names. Next, back to previous page and continue to view the apps allowed ‘Picture-in-picture’ access and remember that list of names. Finally, directly answer the names you remembered. Open the Setting app and go to ‘Special app access’. First, view the apps with ‘Wi-Fi control’ permission and remember the list of app names. Next, view the apps allowed ‘Picture-in-picture’ access and remember that list of names. Finally, open the joplin app, create a note titled ‘App Access Permissions’, and list the names you remembered under two headings: ‘Wi-Fi Control’ and ‘Picture-in-picture’.
078-CheckAndNotePermissions Open the Setting app. Search for ‘Special app access’ and open it. First, check how many apps have ‘Wi-Fi control’ permission and remember the count. Next, check how many apps are allowed to ‘Install unknown apps’ and remember that count. Finally, open the joplin app, create a note titled ‘App Permissions’ that records both counts. Open the Setting app. Search for ‘Special app access’ and open it. First, check how many apps have ‘Wi-Fi control’ permission and remember the count. Next, check how many apps are allowed to ‘Install unknown apps’ and remember that count. Finally, directly answer the counts you remembered. Open the Setting app and go to ‘Special app access’. First, check how many apps have ‘Wi-Fi control’ permission and remember the count. Then, check how many apps are allowed to ‘Install unknown apps’ and remember that count. Finally, open the joplin app and create a note titled ‘App Permissions’ that records both counts.
093-MultiAppPriceComparison Open the Amazon app and search for ‘Bose QuietComfort Ultra Headphones’. Remember the price of it. Then open the Wish app and search for ‘Bose QuietComfort Ultra Headphones’. Remember the price of it. The currencies are different, you need to open the bing app to search for the exchange rate. Finally answer with the name of the app, ‘Amazon’ or ‘Wish’, where the price is lower. Open the Amazon app and search for ‘Bose QuietComfort Ultra Headphones’. Remember the price of it. Then open the Wish app and search for ‘Bose QuietComfort Ultra Headphones’. Remember the price of it. Finally, directly answer the two price. Search for ‘Bose QuietComfort Ultra Headphones’ on both the Amazon and Wish apps, remembering the price and currency from each. If the currencies are different, use the bing app to find the exchange rate to compare them. Directly answer with the name of the app, ‘Amazon’ or ‘Wish’, where the price is lower.
094-MultiAppPriceComparison Open the Amazon app and search for ‘iPhone 16 Pro Max’. Remember the price of it. Then open the Wish app and search for ‘Bose QuietComfort Ultra Headphones’. Remember the price of it. The currencies are different, you need to open the bing app to search for the exchange rate. Finally answer with the name of the app, ‘Amazon’ or ‘Wish’, where the price is lower. Open the Amazon app and search for ‘iPhone 16 Pro Max’. Remember the price of it. Then open the Wish app and search for ‘Bose QuietComfort Ultra Headphones’. Remember the price of it. The currencies are different, you need to open the bing app to search for the exchange rate. Finally, directly answer the two price. Search for ‘iPhone 16 Pro Max’ on both the Amazon and Wish apps, remembering the price and currency from each. If the currencies are different, use the bing app to find the exchange rate to compare them. Directly answer with the name of the app, ‘Amazon’ or ‘Wish’, where the price is lower.
104-ResearchAndNoteF1Principals Open the BBC Sports app, go to the ‘Formula 1’ \rightarrow ‘Constructors” standings. Remember the names and points of the top three teams. Then open the Bing app. For each team, search for their “team principal” name. Finally open the Joplin app, create a note titled “F1 Top 3 Principals”, and write each team with their points and team principal. Open the BBC Sports app, go to the ‘Formula 1’ \rightarrow ‘Constructors” standings. Remember the names and points of the top three teams. Then open the Bing app. For each team, search for their “team principal” name. Finally, directly answer each team with their points and team principal. In BBC Sports app, go to the ‘Formula 1’ \rightarrow ‘Constructors” standings. Remember the names and points of the top three teams. Then, for each team, search in Bing app for their “team principal” name. In Joplin app, create a note titled “F1 Top 3 Principals” listing each team with their points and team principal.
109-AnalyzeMarketAndInvest Open the bing app, search for ‘NVIDIA stock price’, and remember the current stock price. Then open the AP News app, search for ‘NVIDIA earnings report’ and go to the first news (not NVIDIA corp), then remember the reported revenue growth rate. Finally, open the Calculator app, input: stock price * (1 + revenue growth rate), and then calculate the result. Open the bing app, search for ‘NVIDIA stock price’, and remember the current stock price. Then open the AP News app, search for ‘NVIDIA earnings report’ and go to the first news (not NVIDIA corp), then remember the reported revenue growth rate. Finally, directly answer the stock price and revenue growth rate. Open the bing app, search for ‘NVIDIA stock price’, and remember the current price. Then open the AP News app, find the latest news about NVIDIA earnings report, and remember the reported revenue growth rate. Finally, open the Calculator app and calculate: stock price * (1 + revenue growth rate).
110-AnalyzeMarketAndInvest Open the bing app, search for ‘Apple stock price’, and remember the current stock price. Then open the AP News app, search for ‘Apple earnings report’ and go to the first news (not Apple corp), then remember the reported revenue growth rate. Finally, open the Calculator app, input: stock price * (1 + revenue growth rate), and then calculate the result. Open the bing app, search for ‘Apple stock price’, and remember the current stock price. Then open the AP News app, search for ‘Apple earnings report’ and go to the first news (not Apple corp), then remember the reported revenue growth rate. Finally, directly answer the stock price and revenue growth rate. Open the bing app, search for ‘Apple stock price’, and remember the current price. Then open the AP News app, find the latest news about Apple earnings report, and remember the reported revenue growth rate. Finally, open the Calculator app and calculate: stock price * (1 + revenue growth rate).
111-SearchAndSetWallpaper Open the bing app, switch to “image”, and then perform an image search for ‘minimalist black and white wallpaper’. From the results, find the first image that has a portrait (vertical) orientation suitable for a phone wallpaper. Long-press this image to download it. Then, open the Setting app, navigate to wallpaper settings, and change your home screen wallpaper to the image you just downloaded. Open the bing app, switch to “image”, and then perform an image search for ‘minimalist black and white wallpaper’. From the results, find the first image that has a portrait (vertical) orientation suitable for a phone wallpaper. Long-press this image to download it. Then, open the Setting app, navigate to wallpaper settings, and change your home screen wallpaper to the image you just downloaded. Open the bing app and search for ‘minimalist black and white wallpaper’ images. From the results, find the first image that has a portrait (vertical) orientation suitable for a phone wallpaper. Long-press to download this image. Then, open the Setting app, navigate to wallpaper settings, and change your home screen wallpaper to the image you just downloaded.
112-SearchAndSetWallpaper Open the bing app, switch to “image”, and then perform an image search for ‘abstract blue ocean wallpaper’. From the results, find the first image that has a portrait (vertical) orientation suitable for a phone wallpaper. Long-press this image to download it. Then, open the Setting app, navigate to wallpaper settings, and change your home screen wallpaper to the image you just downloaded. Open the bing app, switch to “image”, and then perform an image search for ‘abstract blue ocean wallpaper’. From the results, find the first image that has a portrait (vertical) orientation suitable for a phone wallpaper. Long-press this image to download it. Then, open the Setting app, navigate to wallpaper settings, and change your home screen wallpaper to the image you just downloaded. Open the bing app and search for ‘abstract blue ocean wallpaper’ images. From the results, find the first image that is in a portrait (vertical) format. Long-press and download this image. Afterwards, open the Setting app, navigate to the wallpaper settings, and change your lock screen wallpaper to the downloaded image.
115-FindRecipeAndCalculateCost Open the wikiHow app and search for “how to bake chocolate chip cookies”. Remember the first four ingredients of it. Then open the joplin app and create a checklist named “Cookie Ingredients” with the first four ingredients listed. Finally, open the Calculator app and calculate the total cost, assuming each of the four ingredients costs $3.50. Open the wikiHow app and search for “how to bake chocolate chip cookies”. Remember the first four ingredients of it. Then open the joplin app and create a checklist named “Cookie Ingredients” with the first four ingredients listed. Open the wikiHow app and search for “how to bake chocolate chip cookies”. Create a checklist in the joplin app named “Cookie Ingredients” with the first four ingredients listed. Then, open the Calculator app and calculate the total cost, assuming each of the four ingredients costs $3.50.
116-FindRecipeAndCalculateCost Open the wikiHow app and search for “how to make a pizza”. Remember the first four ingredients of it. Then open the joplin app and create a checklist named “Pizza Ingredients” with the first four ingredients listed. Finally, open the Calculator app and calculate the total cost, assuming each of the four ingredients costs $4.25. Open the wikiHow app and search for “how to make a pizza”. Remember the first four ingredients of it. Then open the joplin app and create a checklist named “Pizza Ingredients” with the first four ingredients listed. Open the wikiHow app and search for “how to make a pizza”. Create a checklist in the joplin app named “Pizza Ingredients” with the first four ingredients listed. Then, open the Calculator app and calculate the total cost, assuming each of the four ingredients costs $4.25.
127-CompareCarsAndResearch Open Cars.co.za app, search for a ‘Volkswagen Polo’, and remember the price of the first result. Then back to homepage and search for a ‘Hyundai i20’ and remember the price of the first result. Open the joplin app to write a note that which car is more expensive and by how much. Finally, open the bing app and search for “Volkswagen Polo vs Hyundai i20 safety rating”. Stay on the search results page. Open Cars.co.za app, search for a ‘Volkswagen Polo’, and remember the price of the first result. Then back to homepage and search for a ‘Hyundai i20’ and remember the price of the first result. Finally, directly answer that which car is more expensive and by how much. Open Cars.co.za app, search for a ‘Volkswagen Polo’, and remember the price of the first result. Then search for a ‘Hyundai i20’ and remember the price of the first result. Open the Calculator app and find the price difference. Open the joplin app to note which car is more expensive and by how much. Finally, open the bing app and search for “Volkswagen Polo vs Hyundai i20 safety rating”. Stay on the search results page.
128-CompareCarsAndResearch Open Cars.co.za app, search for a ‘Ford Ranger’, and remember the price of the first result. Then back to homepage and search for a ‘Toyota Hilux’ and remember the price of the first result. Open the joplin app to write a note that which car is more expensive and by how much. Finally, open the bing app and search for “Ford Ranger vs Toyota Hilux reliability”. Stay on the search results page. Open Cars.co.za app, search for a ‘Ford Ranger’, and remember the price of the first result. Then back to homepage and search for a ‘Toyota Hilux’ and remember the price of the first result. Finally, directly answer that which car is more expensive and by how much. Open Cars.co.za app, search for a ‘Ford Ranger’, and remember the price of the first result. Then search for a ‘Toyota Hilux’ and remember the price of the first result. Open the Calculator app and find the price difference. Open the joplin app to note which car is more expensive and by how much. Finally, open the bing app and search for “Ford Ranger vs Toyota Hilux reliability”. Stay on the search results page.
Table 15: System prompt used by Stamp-GUI.
System Prompt
# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name_for_human": "mobile_use", "name": "mobile_use", "description": "Use a touchscreen to interact with a mobile device, and take screenshots.
This is an interface to a mobile device with touchscreen. You can perform actions like clicking, typing, swiping, etc.
Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions.
The screen’s resolution is 1000x1000.
Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don’t click boxes on their edges unless asked.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:
`key`: Perform a key event on the mobile device.
- This supports adb’s `keyevent`syntax.
- Examples: \"volume_up\", \"volume_down\", \"power\", \"camera\", \"clear\".
`click`: Click the point on the screen with coordinate (x, y).
`long_press`: Press the point on the screen with coordinate (x, y) for specified seconds.
`swipe`: Swipe from the starting point with coordinate (x, y) to the end point with coordinates2 (x2, y2).
`type`: Input the specified text into the activated input box.
`system_button`: Press the system button.
`open`: Open an app on the device.
`wait`: Wait specified seconds for the change to happen.
`answer`: Terminate the current task and output the answer.
`interact`: Resolve the blocking window by interacting with the user.
`terminate`: Terminate the current task and report its completion status.", "enum": ["key", "click", "long_press", "swipe", "type", "system_button", "open", "wait", "answer", "interact", "terminate"], "type": "string"},
"coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=click`, `action=long_press`, and `action=swipe`.", "type": "array"},
"coordinate2": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=swipe`.", "type": "array"},
"text": {"description": "Required only by `action=key`, `action=type`, `action=open`, `action=answer`, and `action=interact`.", "type": "string"},
"time": {"description": "The seconds to wait. Required only by `action=long_press`and `action=wait`.", "type": "number"},
"button": {"description": "Back means returning to the previous interface, Home means returning to the desktop, Menu means opening the application background menu, and Enter means pressing the enter. Required only by `action=system_button`", "enum": ["Back", "Home", "Menu", "Enter"], "type": "string"},
"status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}},
"required": ["action"], "type": "object"}, "args_format": "Format the arguments as a JSON object."}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
# Response format
Response format for every step:
1) Action: a short imperative describing what to do in the UI.
2) Memory: the key information need to memory in the screenshot.
3) A single <tool_call>…</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": <args-json-object>}.
Rules:
- Output exactly in the order: Action, Memory, <tool_call>.
- Be brief: one for Action.
- Do not output anything else outside those three parts.
- If finishing, use action=terminate in the tool call.
Table 16: Prompt used by the dynamic testing planner.
Dynamic Testing Planner Prompt
I have a mobile operation task:
{self.task}
The detailed task guideline is:
{self.guideline}
The calculation part may be difficult, so I am directly giving you some hints:
{self.calculation}
The final answer to input is:
{self.answer}
Now the phone has entered the home page of the task app, as shown in the image below.
Please output exactly one next action. You may choose one of the following four actions:
Click - Please describe the element to click from the perspectives of position, shape, function, or other visual characteristics.
Input - Please describe the element to type into, and also provide the text content to input. This action will first click the target element to activate the input box and then type the text.
Scroll - If you need to scroll down to see more content below, use this action. No extra description is needed.
End - If you have entered the correct answer and seen “Success!”, you may use this action to stop. No extra description is needed.
Please use the following JSON format in your reply:
“‘json
{{"think": "Your detailed reasoning about what to do next, based on the action history, past and current state, task requirement, guideline, and other relevant information", "action": "The next action. Please output it strictly according to the format and requirements above"}}
“‘
Please note:
1. Follow the task guideline strictly. Do not perform exploratory actions based on your own assumptions.
2. The input action already includes one click, so you do not need to click the input box separately before inputting.
3. If a popup appears, try closing it first.
4. Do not try to input the final answer directly. You must first browse all information according to the detailed guideline, and only then input the answer.
5. Before entering the final answer, you must review the action history and screenshots to ensure that all required information for the task has already appeared.
Table 17: Prompt used for memory-accuracy evaluation.
Memory-Acc Evaluation Prompt
I have a task to perform using Chrome on my phone, as follows:
{}
Please note that this task requires memorizing specific information to complete.
I will now provide the content that needs to be memorized for this task, as follows:
{}
I have a mobile AI agent that has completed this task, and it has memorized the following:
{}
Now, please determine whether the information the mobile phone agent has memorized is accurate based on the information above:
1. If the mobile phone agent’s memory completely matches the provided content to be memorized, output “Complete Match”.
2. If the mobile phone agent’s memory partially matches the provided content to be memorized, output “Partial Match”.
3. If the mobile phone agent’s memory does not match the provided content to be memorized at all, output “No Match”.
“Match” means that both expressions convey the same meaning. They do not need to be completely identical. As long as the core content to be memorized, such as numbers or strings, is the same, it is acceptable.
Now, after careful consideration, please directly output your final judgment. You must choose one of “Complete Match”, “Partial Match”, or “No Match” and do not output any other content.
Table 18: Prompt used by the scenario generation agent.
Scenario Generation Agent Prompt
You are a scenario generator.
Your task is to generate a scenario JSON for a simulated mobile web app based on the given spec.
Requirements:
- The scenario must be diverse, realistic, interactive, and verifiable.
- There must be a unique gold answer.
- You may only use values from spec.fixed_values as the source of truth for anything related to the gold answer.
- Do not invent any gold-related values outside spec.fixed_values.
- You may expand copywriting, list items, titles, distracting information, and UI page structure details.
- Output must be strictly valid JSON.
- Output JSON only. No explanation. No markdown.
- All content must be in English only.
- Do not include any Chinese characters anywhere.
Input spec:
{json.dumps(spec, ensure_ascii=False, indent=2)}
You must output a JSON object with exactly the following structure:
{
"meta": {
"seed": <int>,
"platform_type": <string>,
"app_name": <string>,
"theme_keywords": [<string>…]
},
"data": {
"entities": [ … ],
"truth": {
"gold": <string>,
"explain": <string>,
"facts": { … }
}
},
"task": {
"question": <string>,
"must_visit": [<string>…],
"final_output_format": <string>,
"gold": <string>
},
"ui_requirements": {
"tabs": [ … ],
"key_testids": [ … ],
"distraction_interactions": [ … at least 5 items … ],
"pages": [
{
"id": <string>,
"title": <string>,
"purpose": <string>,
"must_show_facts": [<string>…]
}
]
}
}
Requirements for the gold field:
- gold must be a single-line string that can be matched exactly as a string.
- Examples: “123-45A6789”, “coding-course;emotional-intelligence”, “AliceSmith”
- Do not include spaces unless truly required by the format.
- task.gold must be exactly identical to data.truth.gold.
Table 19: Prompt used by the task generation agent.
Task Generation Agent Prompt
You are a task designer.
You will receive a scenario for a simulated mobile web app, including platform information, pages, truth facts, and the gold answer.
You must rewrite it into an executable, gradable, and automation-friendly task specification named task_spec.
Input scenario JSON:
{json.dumps(scn, ensure_ascii=False, indent=2)}
Output requirements:
- Output strictly valid JSON only.
- No explanation.
- No markdown.
- The task must be a memory task:
the user must read information across at least 2 different pages, remember it, and then go to a final submission page to submit the answer.
- The final answer must be unique.
- Grading must use exact string match.
- All content must be in English only.
- Do not include any Chinese characters anywhere.
You must output a task_spec JSON object with exactly the following structure:
{
"platform": {
"app_name": <string>,
"style_keywords": [<string>…],
"tabs": [<string>…]
},
"task": {
"natural_language": <string>,
"guideline": <string>,
"must_visit_pages": [<page_id>…],
"memory_items": [
{
"page_id": <string>,
"fact_key": <string>,
"how_to_find": "Locator hints for automation, preferably referencing data-testid conventions or page module names"
}
],
"final_action": {
"page_id": <string>,
"type": "input_and_submit",
"input_testid": "answer-input",
"submit_testid": "answer-submit",
"required_output_format": <string>,
"gold": <string>
},
"grading": {
"method": "exact",
"gold": <string>,
"pass_regex": <string>
}
},
"ui_contract": {
"required_testids": [<string>…],
"navigation_contract": [
"Example: tab-search/tab-home/tab-orders/tab-me must exist and be clickable for navigation, but must not appear on the submission page"
]
},
"anti_triviality": {
"extra_interactives": [at least 5 irrelevant but reasonable interactive elements],
"notes": <string>
}
}
Hard constraints:
1) task.final_action.gold must be exactly identical to scenario.data.truth.gold.
2) task.grading.gold must be exactly identical to task.final_action.gold.
3) task.grading.pass_regex must match the gold string, preferably using .̂..$.
4) required_testids must include at least:
- all tabs: tab-*
- search input: search-input (if the scenario has a search page)
- final submission: go-submit-answer, answer-input, answer-submit, result
- fact display containers used by memory_items, such as fact-xxx / followers-name / price-sku
5) task.natural_language must clearly describe the task flow, such as which pages to visit and what kind of information to look for, but it must not reveal the actual values that need to be remembered.
6) task.guideline must be a more detailed version of task.natural_language, and must describe the full workflow, all memory targets, and the final submission process in detail.
7) task.guideline must include the exact location and exact content for every memory item, for example:
“Find the code 1234 in My > Orders > History” and
“Find the event code 5678 in Home > Activities > Popular Events”.
Table 20: Prompt used by the webpage generation agent.
Webpage Generation Agent Prompt
You are a web app generator.
You must generate a single-file offline-runnable HTML document with inline CSS and JavaScript for a simulated mobile web app test environment.
Do not use any external network resources, including images, CDNs, web fonts, scripts, or stylesheets.
{fb}
Scenario JSON:
{json.dumps(scn, ensure_ascii=False, indent=2)}
Task_spec JSON:
{json.dumps(task_spec, ensure_ascii=False, indent=2)}
You must implement the following:
1) A mobile app style bottom tab bar using task_spec.platform.tabs, with corresponding unique data-testid values in the form tab-*.
2) Every testid listed in task_spec.ui_contract.required_testids must exist, be unique, and be interactable when appropriate.
3) The navigation behavior described in task_spec.ui_contract.navigation_contract must be implemented.
4) The truth facts in scenario.data.truth.facts must be displayed on the appropriate pages exactly as specified.
Do not alter truth values or truth text.
5) There must be an entry point with data-testid="go-submit-answer" that leads to a dedicated answer submission page containing:
- data-testid="answer-input"
- data-testid="answer-submit"
- data-testid="result"
6) Do not use the <select> element to implement the filter function. If you need to implement the filter function, please use multiple clickable buttons instead.
Submission and validation requirements:
- The one and only correct answer is: {gold}
- Make sure no distractor content creates ambiguity.
- {gold} must remain the unique correct answer under all circumstances.
- The answer input area must include a format hint using the wording “Example: xxx”, where xxx can be any harmless example text.
After clicking submit:
- result must update in a way that supports automated verification.
- If and only if the input exactly equals the gold answer, the page must contain one visible standalone element whose text content is exactly:
Success!
The text must match exactly, including capitalization.
There must be no leading/trailing spaces, no extra punctuation, and no extra newline characters.
- If the input is not exactly equal to gold, then the page must not contain any element whose text content is exactly Success!.
Submission button requirements:
- The button or entry to the submission page must appear prominently in the top-right area.
- Do not place it as a floating control near the bottom or side where it may be covered by fixed elements.
- Preferably make the submission entry visible from every information-bearing page.
Submission page requirements:
- Clicking the submission entry must navigate to a completely separate page or view that is isolated from the previous content.
- The submission page must remove all task_spec.ui_contract.navigation_contract elements, such as bottom tabs, top navigation, floating controls, or other distracting persistent UI.
- The submission page should be kept as clean as possible.
- There must be a back button in the top-left corner that returns to the home page.
Realism and distraction requirements:
- Include at least 5 interactive but task-irrelevant UI elements, such as like buttons, favorite toggles, filters, switches, expandable list items, sort buttons, or menu chips.
- These distractions must not block the main task flow.
- They must not affect the unique correct answer.
Adaptation requirements:
- Include a proper meta viewport tag for mobile rendering.
- The layout must look like a portrait mobile web app.
- Avoid obvious desktop-style page layouts.
Language requirements:
- The entire HTML, including all visible text, labels, placeholders, buttons, tooltips, hints, examples, alerts, validation messages, titles, tab names, and comments embedded in visible content, must be in English only.
- Do not include any Chinese characters anywhere in the HTML.
Output format:
Output JSON only. No explanation.
{
"html": "<!doctype html>….</html>",
"answer": {
"gold": {json.dumps(gold, ensure_ascii=False)},
"facts": {json.dumps(facts, ensure_ascii=False)}
}
}
Table 21: Prompt used by the static inspection agent.
Static Inspection Agent Prompt
You are a static semantic inspector and HTML fixer.
You will receive:
- a scenario containing truth.facts and gold
- a task_spec containing task instructions, workflow, and grading rules
- a single-file HTML source code string
You must inspect the HTML without rendering it.
Your goal is to ensure that a normal user, following the task_spec workflow and reading the required information from the page, will derive one unique answer, and that answer must be exactly equal to gold.
If this is not true, you must directly modify the HTML to fix it.
{fb}
You must output strictly valid JSON only, with exactly the following structure:
{
"ok": <true/false>,
"predicted_answer": <string>,
"uniqueness": {
"is_unique": <true/false>,
"ambiguity_reasons": [<string>…]
},
"evidence": {
"task_interpretation": <string>,
"key_observations": [
{
"page_or_section": <string>,
"what": <string>,
"where_in_html": <string>
}
],
"calculation": <string>,
"key_notes": [<string>, <string>, …]
},
"fix": {
"needed": <true/false>,
"strategy": <string>,
"changes": [
{
"type": <"edit_text"|"edit_number"|"move_block"|"hide_block"|"rename_label"|"other">,
"summary": <string>,
"risk": <"low"|"medium"|"high">
}
]
},
"html": <string>
}
Strict rules:
1) predicted_answer must comply with task_spec.task.final_action.required_output_format, or at least follow the same exact formatting style as task_spec.task.final_action.gold.
2) ok=true if and only if:
- predicted_answer is exactly equal to gold
- uniqueness.is_unique is true
- the task flow is executable and the required information can actually be found in the HTML without contradiction
3) evidence.key_observations must contain enough information for a third party to verify how predicted_answer was obtained without running the page.
4) Use the smallest possible fix strategy, with the following priority:
(a) first adjust distractor values, ordering, labels, or section membership so they no longer affect the task result
(b) second move distractors out of the relevant comparison or calculation area, such as from a ranking list into a recommendation or ad section
(c) avoid changing scenario.truth.facts items whenever possible; if absolutely necessary, mark the change as risk=high and explain why
5) For common bugs involving highest value, top-k, sorting, filtering, or ranking, you must check all candidate items that look comparable, including newly introduced distractors, and ensure they do not change the result away from gold.
6) You may modify the HTML to make the task interpretation clearer, for example by renaming a section title to “Comparison scope for this task: Popularity ranking”, but do not leak the answer in instructional text.
7) evidence.key_notes must list all minimal memory units required for the final answer. If the final answer is formed by combining multiple strings, list each minimal component separately.
8) All output content must be in English only.
9) Do not include any Chinese characters anywhere, including in the repaired HTML.
10) Please ensure that the filter functionality of the webpage is not implemented through the <select> element. It must be implemented through multiple clickable buttons.
scenario.truth.facts:
{json.dumps(facts, ensure_ascii=False, indent=2)}
gold:
{gold}
scenario JSON:
{json.dumps(scn, ensure_ascii=False, indent=2)}
task_spec JSON:
{json.dumps(task_spec, ensure_ascii=False, indent=2)}
HTML source:
—–BEGIN HTML—–
{html}
—–END HTML—–