← 返回阅读笔记 / Back to reading note

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking (WebStep) 原表

Y26-086 原文表格

抽取日期：2026-09-07。来自原文。保留原表行列，未统一实验配置；数值解释见中文分析。

Table 1: Comparison with existing web agent benchmarks. Deterministic : self-hosted deterministic websites. Process eval : Key node uses manual milestones; MDP enables automatic stage decomposition and skill attribution. Hard negative : controlled distractors.
Deterministic Tasks Process-eval Hard negative
Online-Mind2Web (Xue et al., 2025) ✗ 300 ✗ ✗
WebVoyager (He et al., 2024) ✗ 643 ✗ ✗
WebWalkerQA (Wu et al., 2025) ✗ 680 ✗ ✗
AssistantBench (Yoran et al., 2024) ✗ 214 ✗ ✗
WebArena (Zhou et al., 2024) ✓ 812 ✗ ✗
VisualWebArena (Koh et al., 2024) ✓ 910 ✗ ✗
WorkArena++ (Boisvert et al., 2024) ✓ 682 ✗ ✗
Mind2Web-Live (Pan et al., 2024) ✗ 542 Key node ✗
WebStep ✓ 1,800 MDP ✓
Table 2: Aggregate evaluation results. We report terminal success together with process-level metrics from semantic MDP traces: exploration, execution, and information coverage. These metrics reveal behavioral differences not visible from terminal success alone.
Agent Success Rate (%) ↑\uparrow Information ↑\uparrow Steps
Terminal Exploration Execution Coverage (%) GUI Semantic GUI/Semantic\nicefrac{{\mathrm{GUI}}}{{\mathrm{Semantic}}} ↓\downarrow
Fara-7B 35.7 47.3 75.0 76.4 18.9 8.3 2.3
GUI-Owl-1.5-8B 34.4 43.3 78.6 79.5 28.3 10.4 2.7
UI-TARS-1.5-7B 37.1 49.2 73.9 80.3 35.0 14.0 2.5
Qwen3.5-122B 58.2 66.3 86.7 89.1 22.1 9.8 2.3
OpenAI CUA 82.7 89.4 92.4 96.8 19.7 10.0 2.0
Claude CUA 85.3 91.0 93.7 96.3 14.3 9.3 1.5
Table 3: Agent configurations. Input : what the agent receives each turn. Action space : how the agent specifies actions. Max imgs : number of recent screenshots retained in context. Claude CUA follows Anthropic’s DOM-augmented browser harness (ref-based targeting with coordinate fallback), whereas all other agents operate purely from pixels.
Agent Model Backend Input Action Space Max Imgs Temp. Max Tokens
Fara microsoft/Fara-7B vLLM Screenshot Function call (coordinate) 3 0.0 800
UI-TARS ByteDance-Seed/UI-TARS-1.5-7B vLLM Screenshot Thought + Action (coordinate) 5 0.0 2048
GUI-Owl mPLUG/GUI-Owl-1.5-8B vLLM Screenshot Tool call (coordinate) 4 0.0 2048
Qwen3.5 Qwen/Qwen3.5-122B-FP8 vLLM Screenshot Thought + Action (coordinate) 5 0.7 2048
OpenAI CUA GPT-5.4 OpenAI API Screenshot Built-in computer tool (coordinate) – – –
Claude CUA claude-sonnet-4-6 Anthropic API Screenshot + DOM Tool call (ref + coordinate) 3 – –
Table 4: Per-site MDP statistics. Surfaces : distinct UI views (pages, modals). Actions : typed semantic action count. Templates : task template count. Tasks : total task instances. Card fields : attributes visible on list views. Detail fields : attributes requiring detail-page navigation.
Site Domain Surfaces Actions Templates Tasks Card / Detail Fields Decision Object
Mail Productivity 3 36 6 180 10 / 9 Thread
Calendar Productivity 6 14 6 180 8 / 6 Event
Team Chat Productivity 3 24 12 180 6 / 5 Message
Shopping E-Commerce 6 23 6 180 12 / 12 Product
Food Delivery E-Commerce 6 24 15 180 8 / 7 Restaurant / Item
Accommodation Discovery 4 16 12 180 9 / 8 Listing
Housing Discovery 4 16 12 180 8 / 10 Property
Coding Q&A Information 4 18 12 180 7 / 6 Question
Code Repo Collaboration 6 32 12 180 9 / 8 Repository / Issue
Job Network Social 6 26 12 180 7 / 9 Job / Profile
Total – 48 229 105 1,800 84 / 80 –
Table 5: Visibility partition for each site. | 𝒱 card | |\mathcal{V}_{\text{card}}| : attributes visible on list surfaces. | 𝒱 detail | |\mathcal{V}_{\text{detail}}| : attributes visible only on detail surfaces. Surfaces are categorized as Card (list views), Detail (entity detail pages), and Commit (surfaces where task-completing actions are available).
Site Card Surfaces Detail Surfaces Commit Surfaces |𝒱card||\mathcal{V}_{\text{card}}| |𝒱detail||\mathcal{V}_{\text{detail}}|
Mail ThreadList ThreadView ComposeModal, ThreadList, ThreadView 10 9
Calendar WeekView, DayView, MonthView EventDetail EventEditor 8 6
Team Chat ChannelView, SearchResults ChannelView (thread) ChannelView, DirectMessageView 6 5
Shopping SearchResults ProductDetail Checkout, OrderConfirmation 12 12
Food Delivery RestaurantList RestaurantDetail, MenuItemDetail Checkout, OrderStatus 8 7
Accommodation SearchResults ListingDetail Checkout, Reservations 9 8
Housing SearchResults PropertyDetail PropertyDetail, SavedHomes 8 10
Coding Q&A QuestionList QuestionDetail QuestionDetail 7 6
Code Repo RepoList, IssueList, PullRequestList RepoDetail, IssueDetail, PRDetail IssueDetail, RepoDetail 9 8
Job Network JobList, ConnectionList JobDetail, ProfileView JobDetail, Messaging 7 9
Table 6: Skill category definitions. The four exploration categories (Navigate, Search, Filter, Inspect) are global action-type sets; Commit is task-relative, defined per task as the final step of its oracle trajectory.
Category Description
Navigate Actions that move between surfaces or manage the current view without modifying application state. Includes pagination, folder/tab switching, back navigation, closing detail views, and temporal navigation (e.g., next/previous week).
Search Actions that issue or clear search queries to change the visible entity set. Includes keyword search, people search, repository search, and clearing search context.
Filter Actions that narrow or reorder the visible entities within the current result set. Includes applying categorical or range filters, sorting, and clearing filters.
Inspect Actions that reveal additional information about an entity without modifying application state. Includes opening detail views, expanding hidden content, viewing profiles, and selecting entities for closer examination.
Commit The task-completing action: the final step of the task’s oracle trajectory, performed on the target entity—e.g., placing the order, sending the reply, starring the target thread, or, for read-only tasks, opening the target item. State-modifying actions that are not the task’s commit (e.g., adding an item to the cart before placing the order, or filling a form before sending) form the task’s pre-commit sequence; in trajectory analyses these steps are merged into the step that follows them rather than counted as a separate category.
Table 7: Skill taxonomy per site (Part 1 of 2). Skills are organized into five categories: Navigate (surface traversal), Search (query formulation), Filter (attribute filtering and sorting), Inspect (entity viewing), and Commit (task-completing actions).
Site Category Skills
Mail Navigate folder_navigation, pagination, backtrack_navigation
Search search_refinement
Filter filter_application
Inspect thread_inspection, message_expansion
Commit compose_setup, compose_field_entry, email_management, bulk_management, send_commit
Calendar Navigate temporal_navigation, view_switching, backtrack_navigation
Search search_query
Filter calendar_filtering
Inspect event_inspection, information_extraction, comparison, conflict_detection
Commit event_creation, event_editing, field_entry, commit_timing
Shopping Navigate pagination
Search query_formulation
Filter department_filtering, category_filtering, brand_filtering, price_filtering, attribute_filtering, sort_usage
Inspect product_inspection, review_inspection, comparison
Commit option_selection, cart_management, checkout_flow, commit_timing
Accommodation Navigate pagination, search_return
Search query_formulation
Filter property_type_filtering, price_filtering, beds_filtering, amenity_filtering, sort_usage
Inspect listing_inspection, amenity_expansion, comparison
Commit booking_initiation, booking_confirmation, booking_cancellation, checkout_cancellation
Food Delivery Navigate pagination
Search query_formulation
Filter cuisine_filtering, price_filtering, dietary_filtering, attribute_filtering, sort_usage
Inspect restaurant_inspection, menu_item_inspection, comparison
Commit customization_selection, cart_management, checkout_flow, commit_timing
Table 8: Skill taxonomy per site (Part 2 of 2). See Table 6 for category definitions.
Site Category Skills
Housing Navigate pagination
Search query_formulation
Filter price_filtering, beds_filtering, baths_filtering, sqft_filtering, type_filtering, attribute_filtering, sort_usage
Inspect property_inspection, agent_inspection, comparison
Commit save_management, tour_scheduling, agent_contact, saved_homes_review
Coding Q&A Navigate back_navigation, pagination
Search query_formulation
Filter tag_filtering, acceptance_filtering, date_filtering, vote_filtering, sort_usage, answer_sort_usage, filter_reset
Inspect question_inspection, user_inspection, tag_inspection
Commit voting, answer_posting, commenting, answer_acceptance, bookmarking
Code Repo Navigate navigate_to_issues, navigate_to_prs, back_navigation, pagination
Search repo_search
Filter language_filtering, stars_filtering, license_filtering, topic_filtering, repo_attribute_filtering, repo_sort_usage, issue_state_filtering, issue_label_filtering, issue_author_filtering, issue_sort_usage, pr_filtering, pr_sort_usage
Inspect repo_inspection, issue_inspection, pr_inspection
Commit issue_creation, issue_commenting, issue_assignment, issue_labeling, issue_state_change, star_management, commit_timing
Job Network Navigate tab_navigation, pagination
Search job_query_formulation, people_query_formulation
Filter location_filtering, job_type_filtering, remote_filtering, salary_filtering, experience_filtering, sort_usage, filter_clearing
Inspect job_inspection, profile_inspection, job_comparison, profile_comparison
Commit job_application, connection_request, messaging, job_save_action, conversation_management, feed_interaction
Team Chat Navigate channel_navigation, dm_navigation, scroll_navigation, search_result_navigation
Search search_query, search_clear
Filter —
Inspect thread_inspection, member_inspection, thread_close, member_close
Commit send_channel_message, send_thread_reply, send_dm, message_reaction, message_pin, message_edit, message_delete, channel_mute, commit_timing
Table 9: Terminal Success Rate (%) by site. Best per site in bold.
Agent Mail Cal. Shop. Acco. Food Hous. Q&A Code Jobs Chat
Claude CUA 91.7 79.4 82.8 87.2 86.1 79.4 77.8 91.1 84.4 93.3
OpenAI CUA 89.4 73.3 78.9 88.3 81.1 66.7 79.4 88.3 91.1 90.0
Qwen3.5-122B 60.6 52.2 56.1 53.9 73.9 25.6 63.9 65.6 66.7 63.9
Fara 50.0 16.1 37.8 31.1 63.9 13.3 37.2 41.1 40.0 26.1
GUI-Owl 58.3 10.6 12.8 11.7 67.2 20.0 50.6 43.3 35.6 33.9
UI-TARS 71.1 18.9 16.7 8.9 65.6 27.2 46.1 36.7 48.3 31.1
Table 10: Exploration Success Rate (%) by site. Best per site in bold.
Agent Mail Cal. Shop. Acco. Food Hous. Q&A Code Jobs Chat
Claude CUA 91.7 79.4 98.9 93.3 97.8 82.8 85.6 91.1 93.3 93.3
OpenAI CUA 90.0 77.8 95.6 91.7 97.2 70.0 89.4 88.3 92.8 92.2
Qwen3.5-122B 62.8 58.9 71.1 70.0 91.7 27.8 78.9 65.0 67.8 66.7
Fara 53.3 22.2 47.8 52.2 82.2 13.9 52.2 46.1 43.9 55.6
GUI-Owl 58.9 13.3 13.3 47.8 82.2 22.2 60.0 50.6 34.4 47.2
UI-TARS 76.1 26.7 20.0 51.1 82.8 28.9 55.6 41.7 53.3 53.3
Table 11: Execution SR | | Exploration Success (%) by site. Best per site in bold.
Agent Mail Cal. Shop. Acco. Food Hous. Q&A Code Jobs Chat
Claude CUA 100.0 100.0 83.7 93.5 88.1 96.0 89.6 100.0 90.5 100.0
OpenAI CUA 99.4 94.3 82.6 96.4 83.4 95.2 88.8 100.0 97.6 97.6
Qwen3.5-122B 94.7 88.7 78.9 77.0 80.6 80.0 79.6 100.0 96.7 95.8
Fara 93.8 72.5 79.1 59.6 77.0 92.0 70.2 89.2 91.1 47.0
GUI-Owl 99.1 79.2 95.8 24.4 81.8 77.5 84.3 85.7 100.0 71.8
UI-TARS 93.4 70.8 83.3 17.4 77.9 82.7 82.0 88.0 87.5 58.3
Table 12: Informational Coverage at Commit (%) by site. Best per site in bold.
Agent Mail Cal. Shop. Acco. Food Hous. Q&A Code Jobs Chat
Claude CUA 91.3 96.7 100.0 95.5 97.8 100.0 93.3 90.9 98.4 99.1
OpenAI CUA 88.7 97.2 99.6 97.3 98.9 98.9 98.9 90.9 98.4 98.7
Qwen3.5-122B 83.3 94.4 97.2 83.7 93.4 94.4 88.6 68.8 92.7 94.1
Fara 85.0 80.6 93.0 76.6 89.2 62.9 65.3 53.9 76.1 81.7
GUI-Owl 93.9 76.4 74.0 78.5 80.7 96.2 80.6 55.8 72.0 86.8
UI-TARS 84.9 83.1 79.6 81.7 82.5 82.2 81.9 52.5 87.4 86.9
Table 13: Skill invocation rates (%) by site and agent (Part 1 of 2). ”–” = fewer than 20 required episodes. Commit is task-relative: the final task-completing action of each task’s oracle trajectory. Best value per site–skill cell across all six agents in bold.
Agent Site Search Filter Inspect Navigate Commit
Claude CUA Mail 94 – 88 70 79
Cal. – – 100 92 81
Shop. 100 21 100 89 99
Acco. 97 99 100 71 97
Food 70 85 100 63 76
Hous. 99 100 91 69 98
Q&A 86 84 92 73 82
Code 91 100 100 61 93
Jobs 92 100 100 100 81
Chat 100 – 100 100 98
OpenAI CUA Mail 100 – 86 62 66
Cal. – – 100 92 67
Shop. 100 21 100 90 94
Acco. 97 69 100 77 91
Food 77 86 100 68 81
Hous. 98 84 88 73 87
Q&A 91 85 99 43 85
Code 93 90 100 65 83
Jobs 92 97 100 100 95
Chat 100 – 100 84 89
Qwen3.5-122B Mail 86 – 86 76 47
Cal. – – 100 89 54
Shop. 99 35 99 85 72
Acco. 87 85 96 57 44
Food 71 86 99 61 48
Hous. 97 100 87 63 44
Q&A 90 91 93 73 61
Code 82 93 99 39 74
Jobs 99 100 99 100 64
Chat 100 – 99 100 76
Table 14: Skill invocation rates (%) by site and agent (Part 2 of 2). ”–” = fewer than 20 required episodes. Commit is task-relative: the final task-completing action of each task’s oracle trajectory. Best value per site–skill cell across all six agents in bold.
Agent Site Search Filter Inspect Navigate Commit
Fara Mail 97 – 89 41 39
Cal. – – 88 63 21
Shop. 99 71 88 64 35
Acco. 77 83 88 53 6
Food 63 81 96 40 23
Hous. 69 81 71 43 21
Q&A 77 53 70 23 29
Code 71 58 93 32 51
Jobs 88 58 68 94 38
Chat 93 – 88 94 31
GUI-Owl Mail 56 – 93 82 67
Cal. – – 35 35 11
Shop. 100 85 58 46 23
Acco. 91 100 75 22 15
Food 62 90 97 55 26
Hous. 99 100 70 48 38
Q&A 89 91 87 57 60
Code 81 90 97 10 55
Jobs 63 100 79 100 48
Chat 80 – 73 100 54
UI-TARS Mail 95 – 81 73 54
Cal. – – 62 52 17
Shop. 99 56 60 49 13
Acco. 89 100 88 44 3
Food 68 85 99 64 19
Hous. 87 97 74 52 19
Q&A 96 59 84 43 41
Code 87 42 93 42 43
Jobs 94 90 88 98 36
Chat 97 – 84 98 32
Table 15: Aggregate skill invocation rates (%). Each cell shows the fraction of episodes requiring that skill where the agent invoked it. Commit invocation requires the trajectory to end with the task’s commit action. Best per column in bold.
Agent Search Filter Inspect Navigate Commit
Claude CUA 92.9 87.8 97.1 80.9 88.3
OpenAI CUA 94.5 78.9 97.2 79.7 83.8
Qwen3.5-122B 90.9 87.2 95.5 76.3 58.5
Fara 80.4 70.9 82.8 60.3 29.4
GUI-Owl 83.3 94.0 78.6 58.1 39.7
UI-TARS 90.5 77.3 82.9 65.2 27.7
Table 16: Exploration SR (%) by hard negative count (the number of card-level confusables planted by the generator; a designed lower bound). Best per column in bold. HN = = 0 tasks have no identification ambiguity but vary on other difficulty axes, so monotone trends should be read over HN ≥ \geq 1.
Agent HN=0 HN=1 HN=2 HN=3
Claude CUA 90.6 98.7 90.3 89.8
OpenAI CUA 91.7 93.6 87.9 83.5
Qwen3.5-122B 76.4 78.2 65.2 51.4
Fara 60.5 64.1 41.9 32.8
GUI-Owl 59.4 43.6 34.5 32.3
UI-TARS 60.6 53.8 45.6 37.3
Table 17: Exploration SR (%) by oracle trajectory length. Best per column in bold.
Agent 1 2 3 4 5 6 7 8 9 10
Claude CUA 96.3 96.4 89.9 93.9 87.4 84.9 86.7 89.7 95.4 85.9
OpenAI CUA 94.4 96.7 92.9 94.7 80.7 82.4 80.3 87.2 93.1 73.2
Qwen3.5-122B 72.2 73.8 83.2 75.0 67.4 60.4 51.4 53.3 68.2 45.1
Fara 42.6 45.4 61.3 57.6 48.1 44.7 40.5 36.9 42.8 40.8
GUI-Owl 50.0 43.7 57.1 65.2 41.5 43.4 27.6 31.3 38.7 35.2
UI-TARS 50.0 52.3 54.2 62.9 58.5 52.2 40.8 35.4 43.4 38.0
Table 18: Exploration SR (%) by information access level. Best per column in bold. Detail tasks require opening detail pages (non-empty information gap); Filter tasks have no detail gap but their oracle trajectory uses search/filter/sort actions (classified with the shared action taxonomy); Card tasks are solvable from list-level cards alone.
Agent Card Filter Detail
Claude CUA 95.1 98.2 88.3
OpenAI CUA 93.2 96.7 85.8
Qwen3.5-122B 70.4 90.1 59.4
Fara 58.6 72.8 38.9
GUI-Owl 60.5 65.0 35.2
UI-TARS 61.7 61.4 44.2
Table 19: Task template: find_email_extract (Mail). Requires searching, inspecting multiple threads to find one matching body-level content, and starring it.
Field Value
Template find_email_extract
Site Mail
Instruction pattern ”{sender} has sent you several similar emails. Find the one that mentions ’{keyword}’ in its body and {action} it.”
Information gap body, cc, attachments (detail-only fields)
Constraint count 3
Commit action Star / Archive / Label
Concrete instance (gmail_0001)
Instruction ”Priya Patel has sent you several similar emails. Find the one that mentions ’ProjectAlpha006’ in its body and star it.”
Target entity THR-006
Hard negatives THR-019, THR-050 (same sender, different body content)
Oracle trajectory SearchEmails →\to OpenThread(THR-019) →\to CloseThread →\to OpenThread(THR-050) →\to CloseThread →\to OpenThread(THR-006) →\to Star(THR-006)
Optimal length 7 steps
Table 20: Task template: find_product_extract (Shopping). Requires searching, navigating to product detail pages, and adding the correct product to cart based on a detail-only attribute.
Field Value
Template find_product_extract
Site Shopping
Instruction pattern ”Search for {query} in {department}. Find the one with {detail_attr}: ’{value}’ and add it to your cart.”
Information gap specifications, bullet_points, seller, shipping_cost
Constraint count 1–3
Commit action AddToCart
Concrete instance (amazon_0010)
Instruction ”Search for fiction in Books. Find the one with Material: ’Leather’ and add it to your cart.”
Target entity PRD-039 (”Apple Essential Biography”, $12.90, Books, rating 4.7)
Hard negatives Products matching ”fiction” + ”Books” but with different Material
Oracle trajectory Search(”fiction”) →\to OpenProduct(PRD-039) →\to AddToCart
Optimal length 3 steps
Table 21: Task template: search_filter_book (Accommodation). A multi-step booking task requiring search, filter application, sort, listing inspection, and checkout completion.
Field Value
Template search_filter_book
Site Accommodation
Instruction pattern ”Book the {superlative} {property_type} in {location} that has {amenity} for {guests} guests, {dates}.”
Information gap amenities, cancellation_policy, host details
Constraint count 4–6
Commit action ConfirmBooking
Concrete instance (airbnb_0005)
Instruction ”Book the cheapest cabin in Chicago that has a Pool for 3 guests, April 15–19.”
Target entity LST-011
Constraints property_type=Cabin, amenity=Pool, sort=price_asc, location=Chicago
Oracle trajectory Search(”chicago”) →\to SetFilter(property_type, Cabin) →\to SetFilter(amenities, Pool) →\to SortBy(price_asc) →\to ViewListing(LST-011) →\to BookListing(dates, guests) →\to ConfirmBooking
Optimal length 7 steps
Table 22: Semantic trajectory comparison on mail_0001 . All trajectories are shown as semantic MDP actions extracted from the environment trace. The oracle systematically inspects hard negatives before committing; both agents find target item without sufficient information.
Step Oracle Fara UI-TARS
0 SearchEmails("Priya Patel") OpenThread(THR-006) SearchEmails("ProjectAlpha006")
1 OpenThread(THR-019) SearchEmails("ProjectAlpha006") Star(THR-006)
2 CloseThread() OpenThread(THR-006) SwitchFolder(STARRED)
3 OpenThread(THR-050) Star(THR-006)
4 CloseThread()
5 OpenThread(THR-006)
6 Star(THR-006)
7 actions 4 actions (success) 3 actions (success)
Table 23: Semantic trajectory comparison on shopping_0010 . The oracle executes a minimal 3-action sequence. Fara searches but inspects wrong products before adding one to cart (failure-wrong product). UI-TARS repeatedly reformulates queries without reaching the target.
Step Oracle Fara UI-TARS
0 Search("fiction") Search("Books") Search("Books")
1 OpenProduct(PRD-039) ApplyFilter(dept=Books) ApplyFilter(dept=Books)
2 AddToCart() OpenProduct(PRD-009) ClearFilters()
3 GoBack() ApplyFilter(dept=Books)
4 OpenProduct(PRD-027) NewSearch("Leather")
5 GoBack() NewSearch("Leather")
6 OpenProduct(PRD-036) …no further progress
7 AddToCart() ×\times
3 actions (success) 8 actions (failure) 6 actions (failure)