NFL News task: vision agent vs a11y-tree navigation

Same generated website, same keyboard-only user task. The vision agent finishes. The a11y-tree navigation agent reports a navigation-related blocker.

G(open_email_dialog(i,d) → F_[0,500ms] focus_inside(d)) G(close_completed_dialog(d,i) → F_[0,500ms] focus_returns_to_context(i))
Task Email the latest story, close the dialog, then search for Chiefs.
Vision mode keyboard_vision_assisted; screenshot-based context.
A11y-nav mode a11y_tree_navigation_only; compact a11y tree + Orca actions.
Agent reports success

Vision agent

30 steps. Prepared email, closed dialog, searched Chiefs.

Why this is useful

The user-level task can appear complete even when the trace contains modal/focus transitions that require temporal checking.

Agent reports navigation blocker

A11y-tree navigation agent

39 steps. Email/Search controls were reachable but not verifiably operable.

Why this is useful

The a11y-tree-only agent exposes a different failure surface: controls are present in the accessibility navigation path, but activation does not produce the expected dialog/search workflow.