← 返回阅读笔记 / Back to reading note
来源：作者原文。2026-09-07抓取，原始行列，不是统一复跑成绩。
| Env? | # of apps | # task | Avg # task | Reward | Platform | |
| or websites | templates | instances | method | |||
| GAIA | ✗ | n/a | 466 | 1 | text-match | None |
| Mind2Web | ✗ | 137 | 2350 | 1 | None | Desktop Web |
| WebLINX | ✗ | 155 | 2337 | 1 | None | Desktop Web |
| WebVoyager | ✗ | 15 | 643 | 1 | LLM judge | Desktop Web |
| PixelHelp | ✗ | 4 | 187 | 1 | None | Android |
| MetaGUI | ✗ | 6 | 1125 | 1 | None | Android |
| MoTiF | ✗ | 125 | 4707 | 1 | None | Android (Apps+Web) |
| AitW | ✗ | 357+ | 30378 | 1 | None | Android (Apps+Web) |
| AndroidControl | ✗ | 833 | 15283 | 1 | None | Android (Apps+Web) |
| OmniAct | ✗ | 60+ | 9802 | 1 | None | Desktop (Apps+Web) |
| AndroidArena | ✗ | 13 | 221 | 1 | Action match/LLM | Android (Apps+Web) |
| LLamaTouch | ✗ | 57 | 496 | 1 | Screen match | Android (Apps+Web) |
| MiniWoB++ | ✓ | 1 | 114 | HTML/JS state | Web (synthetic) | |
| WebShop | ✓ | 1 | 12k | 1 | product attrs match | Desktop Web |
| WebArena | ✓ | 6 | 241 | 3.3 | url/text-match | Desktop Web |
| VisualWebArena | ✓ | 4 | 314 | 2.9 | url/text/image-match | Desktop Web |
| WorkArena | ✓ | 1 | 29 | 622.4 | cloud state | Desktop Web |
| Mobile-Env | ✓ | 1 | 13 | 11.5 | regex | Android (Apps) |
| B-MoCA | ✓ | 4 | 6 | 1.9 | regex | Android (Apps+Web) |
| MMInA | ✓ | 14 | 1050 | 1 | text-match | Desktop web |
| OSWorld | ✓ | 9 | 369 | 1 | device/cloud state | Desktop (Apps+Web) |
| WindowsAgentArena | ✓ | 11 | 154 | 1 | device state | Desktop (Apps+Web) |
| AgentStudio | ✓ | 9 | 205 | 1 | device state | Desktop (Apps+Web) |
| AndroidWorld | ✓ | 20 | 116 | device state | Android (Apps+Web) |
| Task | Validation code |
|---|---|
| In Simple Calendar Pro, create a calendar event on {event.year}-{event.month}-{event.day} at {event.hour}h with the title ‘{event.title}’ and the description ‘{event.description}’. The event should last for {event.duration} mins. | event_exists(event) |
| Send a text message to {phone_number} with message: {message}. | message_exists(phone_number, message, messaging_db) |
| Create a new drawing in Simple Draw Pro. Name it {file_name}. Save it in the Pictures folder. | file_exists(file_path) |
| Create a timer with {hours} hours, {minutes} minutes, and {seconds} seconds. Do not start the timer. | timer_displays(time, ui_hierarchy) |
| Create a new note in Markor named {file_name} with the following text: {text}. Share the entire content of the note with the phone number {number} via SMS. | (file_exists(file_name, content=text) + message_exists(phone_number, message)) / 2.0 |
| Turn on WiFi and open {app_name}. | (wifi_enabled() + app_launched(app_name))/2.0 |
| Agent | Input | Base model | SRAndroidWorld | SRMobileMiniWoB++ |
|---|---|---|---|---|
| Human | screen | N/A | 80.0 | 100.0 |
| SeeAct (Zheng et al., 2024a) | SoM (screen + a11y tree) | GPT-4 Turbo | 15.5 | 66.1 |
| M3A-Simple | a11y tree | Gemma 2 | 3.4 | 35.5 |
| M3A-Simple | a11y tree | Gemini 1.5 Pro | 14.7 | 55.2 |
| M3A-Simple | a11y tree | GPT-4 Turbo | 19.8 | 67.7 |
| M3A | a11y tree | Gemma 2 | 9.5 | 45.6 |
| M3A | a11y tree | Gemini 1.5 Pro | 19.4 | 57.4 |
| M3A | SoM (screen + a11y tree) | Gemini 1.5 Pro | 22.8 | 40.3 |
| M3A | a11y tree | GPT-4 Turbo | 30.6 | 59.7 |
| M3A | SoM (screen + a11y tree) | GPT-4 Turbo | 25.4 | 67.7 |
| App name | Description | # tasks |
|---|---|---|
| Simple Calendar Pro | A calendar app for creating, deleting, and managing events and appointments. | 17 |
| Settings | The Android system settings app for managing device settings such as Bluetooth, Wi-Fi, and brightness. | 15 |
| Markor | A note-taking app for creating, editing, deleting, and managing notes and folders. | 14 |
| Broccoli - Recipe App | A recipe management app for adding, deleting, and organizing recipes. | 13 |
| Pro Expense | An expense tracking app for adding, deleting, and managing expenses. | 9 |
| Simple SMS Messenger | An SMS app for sending, replying to, and resending text messages. | 7 |
| OpenTracks | A sport tracking app for recording and analyzing activities, durations, and distances. | 6 |
| Tasks | A task management app for tracking tasks, due dates, and priorities. | 6 |
| Clock | An app with stopwatch and timer functionality. | 4 |
| Joplin | A note-taking app. | 4 |
| Retro Music | A music player app. | 4 |
| Simple Gallery Pro | An app for viewing images. | 4 |
| Camera | An app for taking photos and videos. | 3 |
| Chrome | A web browser app. | 3 |
| Contacts | An app for managing contact information. | 3 |
| OsmAnd | A maps and navigation app with support for adding location markers, favorites, and saving tracks. | 3 |
| VLC | A media player app for playing media files. | 3 |
| Audio Recorder | An app for recording and saving audio clips. | 2 |
| Files | A file manager app for the Android filesystem, used for deleting and moving files. | 2 |
| Simple Draw Pro | A drawing app for creating and saving drawings. | 1 |
| Task nickname | Task template | Initialization logic | Success evaluation code |
|---|---|---|---|
| VlcCreatePlaylist | Create a playlist in VLC, titled “{playlist_name}” with the following files, in order: {files} | Create new mpeg files: files + “noise” files that should not be added. Add them to VLC videos folder. | execute_sql(vlc_query) == files |
| RecipeAddMultiple RecipesFromImage | Add the recipes from recipes.jpg in Simple Gallery Pro to the recipe app. | Write a receipt file with recipes to Simple Gallery. | sql_rows_exist(expected_recipes) |
| MarkorEditNote | Edit {file_name} in Markor. {file_operation}. | Generate file with starting content, along with “noise” files not relevant to goal. Note: file_operation can be to add a footer, header, or update note content. | file_exists(file_name, content=expected_content) |
| ExpenseAddSingle | Add the following expenses into pro expense: {expense_csv} | Add to the app’s SQLite database the expense that should be deleted, along with “noise” expenses that should not be deleted. | sql_rows_exist(expense_obj) |
| SimpleCalendarDelete EventsOnRelativeDay | In Simple Calendar Pro, delete all events scheduled for this {day_of_week}. | add to the app’s SQLite database calendar events on specified day, along with “noise” events that should not be deleted. | !sql_rows_exist(expected_events) |
| FilesDeleteFile | Delete the file {file_name} from the Android filesystem located in the {subfolder} folder within the sdk_gphone_x86_64 storage area. | Generate specified file, along with “noise” files that should not be deleted. | !file_exists(file_name) |
| SportsTrackerActivities CountForWeek | How many {category} activities did I do this week in the OpenTracks app? Express your answer as a single integer. | add to the app’s SQLite database activities for the specified category, along with “noise” activities. | int(agent_response) == expected_count |