BLV AP task gallery

Keyboard-only reproductions of accessibility-property tasks, with representative Agent–LTL comparisons followed by the original paired-agent run gallery.

6 Agent–LTL comparison demos 10 accessibility properties 20 runs 2 modes per property Batch: ap-gallery-full20-flow-grounded-20260808

Representative Agent vs. LTL demos

All six examples use two columns. The first three compare matched Agent and LTL reproductions. The next three compare an Agent task run from the 485-run evaluation with an independent 30-minute LTL-only exploration of the same website and generation condition. Use the timestamp buttons to jump to the evidence; explanatory callouts appear over the video during playback.

1Execution Agent acts

It performs a keyboard task and saves each action and resulting focus/page state.

2History is saved

The replay shows those steps. A video frame by itself is not yet an analyzer verdict.

3Agent analyzer labels afterward

It reads the saved history, applies a taxonomy rule, and returns an error type such as N2, W3, or FD1.

4LTL checks registered rules

Passive LTL watches Agent actions; LTL-only chooses its own exploration. Both can report only a formal AP whose trigger and outcome are provable.

Pairing scope for the first three demos: the two replays share the same AP, website, and task seed, but they are separate trajectories from separate batches. They are paired for behavioral comparison, not presented as two views of one synchronized episode. Agent replays come from ap-gallery-full20-flow-grounded-20260808; LTL replays come from the 30-minute targeted campaign.

What an “Agent error” video label means: the execution Agent did not diagnose the bug in real time. The post-hoc analyzer later read the task/final summary and up to 60 reduced history steps—action, result, URL, and focus before/after—without seeing LTL output. For presentation, this page pins the analyzer’s error type back onto the step that supplied its evidence. The first three targeted gallery runs did not persist a blind-analyzer output, so their labels are transparent applications of the same trace-to-taxonomy template; the last three use real persisted blind-analyzer results from Vision-485.

AP-FM1-01 · Focus loses meaningful continuation

The methods agree that focus handling is broken, but their primary evidence is different: the Agent explains the modal-opening experience, while LTL proves a later control-invalidation contract violation.

Same AP label · different evidence

Execution Agent replay

Complete task replay

MP4

Complete 49-step replay · 01:19 · before/after frame for every Agent step

  1. Enter opens “Review and book,” but focus remains on the background Reserve button.
  2. Repeated Tab moves through other results and footer links behind the visible dialog.
  3. Focus finally reaches the dialog’s Close button after traversing the background.
  4. The raw Agent trace later records document focus when the notification control disappears—closer to LTL’s witness, although the Agent’s final explanation emphasizes modal entry.
Original Agent task prompt · expand
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-FM1-01 - Focused control invalidated; focus loses meaningful continuation.

Complete a realistic mock booking.

Source workflow steps: Search any travel category and choose Select, Reserve, or Choose on a result. Review the item, enter traveler name and confirmation email, and click Confirm booking. A unique `EXP-` confirmation notification appears, and the booking is then listed under Upcoming trips on `/trips`.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.

LTL checker

Complete exploration replay

MP4

Complete 30-minute exploration · original-timing reconstruction from all 3,935 screenshots

  1. LTL has continuously monitored the formal AP; it now observes keyboard activation of “Dismiss notification.”
  2. The activated control is removed and, after the 500 ms settle window, focus is the document rather than a connected, rendered, accessibility-exposed element.
  3. The checker emits AP_FM1_01_LOCAL_CONTROL_INVALIDATION_RETAINS_EXPOSED_FOCUS with the action, removed-control reason, settled focus, and screenshot in one witness.
Actual checker code · AP-FM1-01

The rule applies after trusted Enter/Space invalidates the focused control. It reports a violation unless settled focus is a connected, rendered, accessibility-exposed element.

export function fm1Applicable(transaction) {
  return fm1OpportunityObserved(transaction)
    && fm1EvidenceComplete(transaction)
    && transaction.observation.contextStable === true
    && transaction.post.controlInvalidation.invalidated === true;
}

export function fm1SettledFocusIsSafe(transaction) {
  const focus = transaction?.post?.settledFocus;
  return focus?.kind === "element"
    && focus.connected === true
    && focus.rendered === true
    && focus.accessibilityExposed === true;
}

export function fm1FocusResetViolation(transaction) {
  return fm1Applicable(transaction)
    && !fm1SettledFocusIsSafe(transaction);
}

export const AP_FM1_01_LOCAL_CONTROL_INVALIDATION_RETAINS_EXPOSED_FOCUS =
  always(() =>
    fm1LocalInvalidationFocusContractHolds(fm1ActivationObservation.current));

Why it fires here: Dismiss notification is removed, but settledFocus.kind === "document", so fm1SettledFocusIsSafe is false.

Source: online-v2/specs/atomic/properties/focus/ap-fm1-01.ts

How the Agent analyzer reaches the finding

The analyzer would separate two focus events that the execution Agent’s final narrative grouped under AP-FM1-01.

Trace evidence
Modal-entry trigger: step 24 Enter keeps focus Reserve → Reserve; steps 25–35 move through other result and footer controls before step 36 reaches Close booking. Strict FM1 trigger: step 42 Enter moves Confirm booking → body/document. The final summary describes the background traversal. Raw-trace-only corroboration records modal changes 0→1 and 1→0, plus focus_on_document_after_keyboard_action at step 42.
Taxonomy rule
FM1 · Focus reset after action“After an action such as submit or close, focus jumps back to page top or an unrelated element.”The earlier modal-entry path is closer to FM2 · Off-target focus drift.
Analyzer decision
Step 42 directly satisfies FM1 because a submit/close transition leaves focus on the document. Steps 24–36 are also a website focus failure, but fit FM2 more precisely: focus drifts through background content rather than resetting after submit/close.
Relation to LTL
The crosswalk maps AP-FM1-01 → FM1. LTL is narrower: trusted activation invalidates the focused control, then settled focus must remain on a connected, rendered, exposed element. Its witness is a different action—Space on Dismiss notification, followed by removal and document focus. Same broad FM1 family, different primary event; not exact event agreement.

AP-M1-03 · Escape does not close the active dialog

This is the cleanest agreement case: both methods exercise Escape on the NFL player and observe that the same dialog remains open.

Exact behavioral agreement

Execution Agent replay

Complete task replay

MP4

Complete 17-step replay · 00:28 · before/after frame for every Agent step

  1. Enter opens the full-screen player; “NOW PLAYING” is visible.
  2. Escape is pressed, but modal_count stays 1 and the player remains visible.
  3. The Agent continues testing: Pause, Resume, and the explicit Close control work, isolating the defect to Escape handling.
Original Agent task prompt · expand
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-03 - Escape does not close exact active dialog.

Watch the halftime show from the most recent Super Bowl

Source workflow steps: - Visit `/videos` from the Videos navigation item. - Select the “Halftime Show” category, then choose the featured “Super Bowl LVIII Halftime Show: Usher’s full performance” play control. - A full-screen mock player opens and visibly reports “NOW PLAYING”; its pause/resume and close controls work.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.

LTL checker

Complete exploration replay

MP4

Complete 30-minute exploration · original-timing reconstruction from all 3,790 screenshots

  1. A trusted Escape transaction begins while the NFL player dialog is active.
  2. At the 500 ms deadline, sameDialogConnected=true and sameDialogVisible=true.
  3. The checker emits AP_M1_03_ESCAPE_CLOSES_EXACT_ACTIVE_DIALOG; “exact” matters because it tracks the original dialog, not merely whether some modal exists.
Actual checker code · AP-M1-03

The rule starts on trusted Escape and checks the identity of the original active dialog after the 500 ms deadline.

export function m1EscapeDialogApplicable(transaction) {
  return transaction?.action?.key === "Escape"
    && transaction.action.trusted === true;
}

export function m1EscapeDidNotCloseDialogViolation(transaction) {
  return m1EscapeDialogApplicable(transaction)
    && m1EscapeDialogEvidenceComplete(transaction)
    && transaction.post.sameDialogConnected === true
    && transaction.post.sameDialogVisible === true;
}

export const AP_M1_03_ESCAPE_CLOSES_EXACT_ACTIVE_DIALOG =
  always(() =>
    m1EscapeDialogContractHolds(m1EscapeDialogObservation.current));

Why it fires here: after Escape, the saved exact player dialog is still both connected and visible.

Source: online-v2/specs/atomic/properties/dialog/ap-m1-03.ts

How the Agent analyzer reaches the finding

Here the broad taxonomy rule and the exact LTL property point to the same Escape event.

Trace evidence
Step 8 Enter on Play Super Bowl LVIII halftime show keeps focus on the Play control; step 9 Escape again leaves focus there. Steps 10–12 Tab to Watch now, Close video, and Pause video; step 16 Enter on Close moves focus to the document. The final summary says Escape failed while Pause, Resume, and Close worked. Raw-trace-only corroboration records modal count 1→1 and modal_still_present_after_escape.
Taxonomy rule
M1 · Modal-dismissal struggle“An unexpected modal intercepts focus and its dismiss sequence is unclear.”
Analyzer decision
The normal dismissal key does not close the modal, so the user must find a separate Close control. The later successful explicit close attributes the problem to website Escape handling rather than a dead browser session or an entirely inoperable dialog.
Relation to LTL
The crosswalk maps AP-M1-03 → M1. LTL formalizes one exact slice: after trusted Escape on the exact active dialog, that same dialog must no longer be connected and visible after 500 ms. Both methods point to the same Escape and player event. Exact behavioral agreement; Agent taxonomy is broader, while LTL supplies the bounded proof.

AP-M1-05 · A newly opened modal does not receive focus

Both methods identify the same opening transition. The Agent then demonstrates the keyboard consequence, while LTL captures the minimal formal witness immediately.

Semantic agreement · taxonomy mismatch

Execution Agent replay

Complete task replay

MP4

Complete 16-step replay · 00:26 · before/after frame for every Agent step

  1. The player opens, but focus remains on the background “Play Super Bowl LVIII halftime show” button.
  2. Tab moves to another background “Watch now” button instead of Pause or Close in the modal.
  3. Escape also fails; Shift+Tab returns to the original background Play control. The Agent reports that the player cannot be operated by keyboard.
Original Agent task prompt · expand
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-05 - New modal does not receive focus.

Watch the halftime show from the most recent Super Bowl

Source workflow steps: - Visit `/videos` from the Videos navigation item. - Select the “Halftime Show” category, then choose the featured “Super Bowl LVIII Halftime Show: Usher’s full performance” play control. - A full-screen mock player opens and visibly reports “NOW PLAYING”; its pause/resume and close controls work.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.

LTL checker

Complete exploration replay

MP4

Same complete NFL LTL exploration as AP-M1-03 · one run can yield multiple AP witnesses

  1. LTL observes trusted Enter on the halftime-show Play button and records that zero modals existed before activation.
  2. One exact new modal is identified, but focusInsideExactNewModal=false after the 500 ms deadline.
  3. The checker immediately emits AP_M1_05_NEW_EXACT_MODAL_RECEIVES_FOCUS, without needing a long Tab traversal to establish the violation.
Actual checker code · AP-M1-05

The rule applies when trusted Enter/Space opens at least one modal. Complete evidence must identify exactly one new modal and determine whether focus entered it.

export function m1NewModalFocusApplicable(transaction) {
  return ["Enter", " "].includes(transaction?.action?.key)
    && transaction.action.trusted === true
    && Number.isInteger(transaction?.post?.newModalCount)
    && transaction.post.newModalCount > 0;
}

export function m1NewModalDidNotReceiveFocusViolation(transaction) {
  return m1NewModalFocusApplicable(transaction)
    && m1NewModalFocusEvidenceComplete(transaction)
    && transaction.post.focusInsideExactNewModal !== true;
}

export const AP_M1_05_NEW_EXACT_MODAL_RECEIVES_FOCUS =
  always(() =>
    m1NewModalFocusContractHolds(m1NewModalFocusObservation.current));

Why it fires here: newModalCount === 1 and the exact modal is identified, but focusInsideExactNewModal === false.

Source: online-v2/specs/atomic/properties/dialog/ap-m1-05.ts

How the Agent analyzer reaches the finding

The history clearly supports the LTL event, but the broad Agent taxonomy has no exact “new modal must receive focus” class.

Trace evidence
Step 12 Enter on Play Super Bowl LVIII halftime show leaves focus Play → Play; step 13 Tab moves to background Watch now · 13:08, not Pause or Close; step 14 Escape keeps focus on Watch now; step 15 Shift+Tab returns to Play. The final summary says the player is visible but its controls remain unreachable. Raw-trace-only corroboration records modal count 0→1 at step 12 and 1 after Escape.
Taxonomy rule
FM2 · Off-target focus drift“Tab or heading navigation overshoots the intended element and focus lands in an adjacent unrelated section.”The failed Escape can additionally support broad M1 · Modal-dismissal struggle, but M1 does not encode the modal-entry focus obligation.
Analyzer decision
The new modal never takes focus and forward/backward navigation remains in the page behind it, satisfying FM2 and explaining why visible player controls are unreachable. Escape failure is a second M1-like symptom, not the proof of modal-entry failure.
Relation to LTL
The fixed crosswalk maps AP-M1-05 → M1, but the Agent evidence naturally maps to FM2. LTL checks the exact opening transaction: newModalCount=1, exactNewModalIdentified=true, and after 500 ms focusInsideExactNewModal=false. Same concrete focus event, but semantic agreement is hidden by the AP-to-taxonomy mismatch.

Agent-only findings from Vision-485, with LTL-only comparison

These left-hand videos are ordinary task runs, not AP-targeted reproductions. Their post-hoc Agent analyzer findings are real persisted outputs. Each right-hand video is a separate 30-minute LTL-only exploration of the same website and generation condition. The two columns are intentionally not synchronized: one follows a user task, while the other explores independently.

What “LTL-only did not find this Agent bug” means: we compare the specific control and behavior, not whether LTL reported anything else on the website. For N2 the formal rule measures the wrong navigation segment; for W3 the formal run finds a different checkbox bug but cannot judge the Agent’s Explore button; for FD1 the checker sees the same dead-link controls but has no registered FD1 rule.

N2 · Thirty Tabs to the first Compare control

The Agent analyzer follows the task target across many history steps. LTL’s AP-N2-01 instead counts only DOM keyboard stops before the page’s main region, so it never evaluates this long path inside main content.

Agent N2 · LTL-only misses this path

Agent task run · analyzed afterward

Apartments · Compare homes side by side

MP4

Complete 43-step condensed replay · 01:10 · one end-state frame for every saved Agent step

  1. Step 7 begins the uninterrupted Tab sequence through header controls.
  2. Step 12 reaches the Location textbox at y=-2017, an off-screen stop on the way to the task target.
  3. Step 23 is still traversing filter controls; the target is not yet reachable.
  4. Step 36 finally reaches the first home’s comparison checkbox after 30 consecutive Tabs.
Original Agent task prompt · expand
Compare homes side by side.

Source workflow steps: On Rent results, check Compare on up to three homes and choose Compare homes in the floating bar. The comparison page shows monthly rent, bedrooms, bathrooms, type, rating, and virtual-tour availability; homes can be removed or opened.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.

Independent LTL-only exploration

Apartments G0 · same generated condition

MP4

Complete 30-minute LTL-only exploration · original timing · all 4,056 screenshots

  1. AP-N2-01 sees only 7 keyboard stops before <main>; its required threshold is 20.
  2. After independent exploration, AP-N2-01 ends NOT_APPLICABLE. It never measures the long in-main route to the task’s Compare checkbox.
  3. LTL reports other APs on Apartments, but no witness for this N2 task-path failure.

How the Agent analyzer finds N2, and what LTL-only measures instead

Persisted blind-analyzer result: N2 · Verbose linear traversal, website attribution, high confidence.

Trigger and rule
Steps 7–36 contain 30 consecutive Tab actions while the task is trying to reach a Compare checkbox.N2: “User reaches target via brute-force Tab/Down across 100s of elements due to missing shortcuts.”
History evidence read by Agent
The saved focus path crosses header links, Location and Search, the full filter sidebar, breadcrumb, Save search, Sort results, image/favorite/title controls, and only then the first Compare checkbox. No skip or direct results-navigation control appears in that path.
Why the Agent classifies N2
The analyzer uses the task target and the whole action sequence: the user had to brute-force nearly the entire interface to reach the first relevant control. It can call this a broad N2 usability failure even though the taxonomy description is not implemented as a fixed numeric threshold.
LTL rule and result
AP-N2-01 is narrower: it applies only when at least 20 visible enabled keyboard stops occur before <main>, then checks for a named exact fragment bypass. The task trajectory’s passive observation was preMainKeyboardStopCount=7 and exactBypassControls=[]. The independent 30-minute LTL-only run reaches the same formal outcome: AP-N2-01 = NOT_APPLICABLE. Neither run measures the in-main route to Compare.
Coverage gap
Task-target gap. Agent measures the actual route to “Compare” inside main content; LTL measures a structural pre-main proxy. Both results are internally consistent, but only the Agent represents the failure seen in this video.

W3 · Enter expands Explore, Space does nothing

The Agent analyzer compares two adjacent actions on the same control. LTL observes the control but refuses to judge it because the generic button has no admitted exact-effect oracle.

Agent Explore bug · LTL-only misses it

Agent task run · analyzed afterward

Udemy · Unreal Engine course search

MP4

Complete 37-step condensed replay · 01:01 · one end-state frame for every saved Agent step

  1. Step 25: Enter on Explore changes expanded=false → expanded=true.
  2. Step 26: Space on the same focused Explore button leaves expanded=true; URL, focus, and control-state hash do not change.
Original Agent task prompt · expand
Find Unreal Engine game development courses with 17+ hours.

Source workflow steps: Search “Unreal Engine game development” and choose Duration “17+ hours.” Expected result: the 24.5-hour Complete Game Development Bootcamp and the 19-hour Blueprints RPG course are shown.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.

Independent LTL-only exploration

Udemy G0 · same generated condition

MP4

Complete 30-minute LTL-only exploration · original timing · all 4,083 screenshots

  1. LTL-only reaches Explore, but records generic_button_effect_undeclared; the transaction is excluded from a violation claim.
  2. Space on Explore is again observed but not admitted because no exact expected effect can be proved.
  3. LTL later fires AP-W3-01 on a different control—“Beginner favorites,” a checkbox whose checked state fails to change.
  4. Comparison result: same formal family, different bug. The Explore Enter/Space mismatch found by Agent is still absent from LTL’s reported witnesses.

How the Agent analyzer finds W3, and why LTL-only misses Explore

Persisted blind-analyzer result: W3 · Activation-mode mismatch, website attribution, high confidence.

Trigger and rule
The Agent tests two standard activation keys on the same focused Explore button.W3: “A widget expects Space but the screen reader sends Enter, or vice versa, so the control appears non-responsive.”
History evidence read by Agent
Step 25 records Enter and focus state Explore expanded=false → expanded=true. Step 26 records Space on the same control and expanded=true → expanded=true. This within-control contrast isolates the mismatch from a general page failure.
Why the Agent classifies W3
One supported activation mode visibly changes the widget while the other has no result. Because the comparison is adjacent and uses the same named control, the analyzer can make a high-confidence behavioral judgment from history.
LTL rule and result
AP-W3-01 requires an admitted role/key transaction with a provable exact postcondition within 500 ms. On Explore, both the task trajectory and the independent LTL-only run record role=button, admitted=false, admissionReason=generic_button_effect_undeclared, and effectKind=null. The independent run later emits AP-W3-01 for a different “Beginner favorites” checkbox whose checked state fails to change. Thus LTL finds a W3-family bug on Udemy, but not the Explore activation mismatch shown at left.
Coverage gap
Oracle-admission gap for this control. Agent infers Explore’s expected behavior by comparing Enter and Space in adjacent history steps. LTL can prove the checkbox transition elsewhere, but intentionally declines to guess what exact effect the generic Explore button should have.

FD1 · Four links accept Enter but provide no feedback

The Agent analyzer uses repeated failed activations plus successful neighboring links as a control comparison. Current LTL has no direct FD1 property, so this behavior is outside its reported scope.

Agent FD1 · LTL-only has no FD1 rule

Agent task run · analyzed afterward

Agoda · Responsive discovery navigation

MP4

Complete 44-step condensed replay · 01:12 · one end-state frame for every saved Agent step

  1. Step 31: Enter on Help center leaves URL /deals and focus on Help center.
  2. Step 33: Manage bookings behaves the same way.
  3. Step 35 is the control comparison: Cancellation options successfully changes /deals → /help.
  4. Steps 39 and 41 repeat the no-result pattern on Things to do and Airport car rentals.
  5. Step 43 confirms the browser is still functional: Sign in or register changes /deals → /account.
Original Agent task prompt · expand
Responsive discovery navigation

Source workflow steps: Use trending destination cards, product tabs, feature cards, header navigation, or footer navigation to move between every product and support route. The mobile menu exposes the same destinations at narrow screen widths.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.

Independent LTL-only exploration

Agoda G0.5 · same generated condition

MP4

Complete 30-minute LTL-only exploration · original timing · all 4,131 screenshots

  1. LTL-only activates Help center and Airport car rentals, but records only excluded link_destination_not_narrowly_comparable candidates.
  2. It also reaches Manage bookings; again the checker cannot prove the application-specific destination or feedback contract.
  3. Things to do produces the same excluded candidate. These are observations, not LTL violations.
  4. Comparison result: LTL-only encounters the same named controls, but the 21-property registry has no FD1 obligation, so it cannot emit the Agent’s “no action feedback” finding.

How the Agent analyzer finds FD1, and why LTL-only cannot report it

Persisted blind-analyzer result: FD1 · No action feedback, website attribution, high confidence.

Trigger and rule
Enter is pressed on four focused controls exposed as links: Help center, Manage bookings, Things to do, and Airport car rentals.FD1: “No textual or programmatic feedback confirms an action outcome, so the user must re-traverse the page to verify it.”
History evidence read by Agent
At steps 31, 33, 39, and 41, both URL and focus remain unchanged after Enter. The same trace also contains successful neighboring activations: Cancellation options changes /deals → /help, Limited Time Offers returns to /deals, and Sign in or register changes /deals → /account.
Why the Agent classifies FD1
The repeated no-result pattern is not inferred from one quiet frame. Successful adjacent links establish what observable feedback normally looks like in this workflow, while four named links consistently provide none. Raw trace fields corroborate the analyzer-visible evidence with an unchanged control-state hash and activation_no_observed_effect.
LTL rule and result
The current 21-property registry has no direct FD1 AP, so no formal rule can emit an FD1 witness. In the independent 30-minute replay, LTL-only actually reaches Help center, Manage bookings, Things to do, and Airport car rentals. It records AP-W3-01 candidates with link_destination_not_narrowly_comparable, then excludes them from claims because the expected application destination or feedback is not provable. Seeing the same controls is therefore not the same as detecting FD1.
Coverage gap
Missing-property gap. Agent can use task context and contrastive history to judge whether feedback was expected. LTL currently has no registered obligation representing that generic expectation.

AP-M1-01: Dialog lacks usable accessible name

A11y-tree navigation only

ap-m1-01-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
14
prompt_tokens
135570
prompt_cached_tokens
48384
completion_tokens
3009
total_tokens
138579

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/imdb/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
required
flow_index
21
flow_title
Give Terminator 2: Judgment Day a rating of 10.
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-01 - Dialog lacks usable accessible name.

Give Terminator 2: Judgment Day a rating of 10.

Source workflow steps: Search “Terminator 2,” open the title, select Rate, and choose 10 in the rating dialog. The title page updates Your Rating to 10/10 and shows a confirmation status.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Completed using keyboard and Orca navigation only. Terminator 2: Judgment Day now shows Your Rating as 10/10. The rating dialog was announced without a usable accessible name, consistent with AP-M1-01.

Keyboard + vision assisted

ap-m1-01-vision

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
22
prompt_tokens
124122
prompt_cached_tokens
0
completion_tokens
4011
total_tokens
128133

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/imdb/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
required
flow_index
21
flow_title
Give Terminator 2: Judgment Day a rating of 10.
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-01 - Dialog lacks usable accessible name.

Give Terminator 2: Judgment Day a rating of 10.

Source workflow steps: Search “Terminator 2,” open the title, select Rate, and choose 10 in the rating dialog. The title page updates Your Rating to 10/10 and shows a confirmation status.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Completed using keyboard-only actions. Terminator 2: Judgment Day now shows Your Rating: 10/10, with the confirmation status: “Your rating for Terminator 2: Judgment Day: 10/10.”

AP-M1-03: Escape does not close exact active dialog

A11y-tree navigation only

ap-m1-03-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
8
prompt_tokens
78039
prompt_cached_tokens
27648
completion_tokens
2221
total_tokens
80260

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/nfl/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
required
flow_index
1
flow_title
Watch the halftime show from the most recent Super Bowl
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-03 - Escape does not close exact active dialog.

Watch the halftime show from the most recent Super Bowl

Source workflow steps: - Visit `/videos` from the Videos navigation item. - Select the “Halftime Show” category, then choose the featured “Super Bowl LVIII Halftime Show: Usher’s full performance” play control. - A full-screen mock player opens and visibly reports “NOW PLAYING”; its pause/resume and close controls work.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Completed the keyboard-only workflow. The full-screen mock player for “Super Bowl LVIII Halftime Show: Usher’s full performance” is open and reports “NOW PLAYING.” The dialog also exposes “Close video” and “Pause video” controls.

Keyboard + vision assisted

ap-m1-03-vision

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
17
prompt_tokens
91709
prompt_cached_tokens
0
completion_tokens
3388
total_tokens
95097

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/nfl/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
required
flow_index
1
flow_title
Watch the halftime show from the most recent Super Bowl
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-03 - Escape does not close exact active dialog.

Watch the halftime show from the most recent Super Bowl

Source workflow steps: - Visit `/videos` from the Videos navigation item. - Select the “Halftime Show” category, then choose the featured “Super Bowl LVIII Halftime Show: Usher’s full performance” play control. - A full-screen mock player opens and visibly reports “NOW PLAYING”; its pause/resume and close controls work.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Workflow completed using keyboard only. Selected “Halftime Show,” opened “Super Bowl LVIII Halftime Show: Usher’s full performance,” and verified the full-screen mock player displayed “NOW PLAYING.” Confirmed AP-M1-03: Escape did not close the active dialog. The pause control changed the player to “PAUSED” with “Resume video,” resume restored playback, and the “Close video” control successfully closed the player.

AP-M1-04: Dialog closes but focus is not restored to the invoker

A11y-tree navigation only

ap-m1-04-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
8
prompt_tokens
79007
prompt_cached_tokens
41472
completion_tokens
1859
total_tokens
80866

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/nfl/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
additional
flow_index
8
flow_title
Use global search
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-04 - Dialog closes but focus is not restored to the invoker.

Use global search

Source workflow steps: - Activate Search in the header (or press `/`), type a team, player, game, or video query, and choose a matching result. The app navigates directly to the appropriate working route.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Workflow completed and AP-M1-04 was reproduced. I opened global Search from the header, entered “Chiefs,” and activated “Super Bowl LIV 2020 · Chiefs 31, 49ers 20.” The app navigated to `/scores?season=2020`, showing Super Bowl LIV with Chiefs 31 and 49ers 20. After the search dialog closed, focus was on the document body rather than restored to the Search invoker.

Keyboard + vision assisted

ap-m1-04-vision

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
15
prompt_tokens
77990
prompt_cached_tokens
0
completion_tokens
2360
total_tokens
80350

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/nfl/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
additional
flow_index
8
flow_title
Use global search
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-04 - Dialog closes but focus is not restored to the invoker.

Use global search

Source workflow steps: - Activate Search in the header (or press `/`), type a team, player, game, or video query, and choose a matching result. The app navigates directly to the appropriate working route.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Completed the global search workflow using only keyboard actions. Selected “Tom Brady — Career passing statistics,” which navigated to `/players?q=Tom%20Brady`; the Tom Brady player profile is visible. AP-M1-04 was confirmed: after the search dialog closed, focus was on the page body rather than restored to the Search invoker.

AP-M1-05: New modal does not receive focus

A11y-tree navigation only

ap-m1-05-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)failed
steps
8
prompt_tokens
77952
prompt_cached_tokens
34560
completion_tokens
2073
total_tokens
80025

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/nfl/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
required
flow_index
1
flow_title
Watch the halftime show from the most recent Super Bowl
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-05 - New modal does not receive focus.

Watch the halftime show from the most recent Super Bowl

Source workflow steps: - Visit `/videos` from the Videos navigation item. - Select the “Halftime Show” category, then choose the featured “Super Bowl LVIII Halftime Show: Usher’s full performance” play control. - A full-screen mock player opens and visibly reports “NOW PLAYING”; its pause/resume and close controls work.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
NAVIGATION_RELATED_ERROR: The Super Bowl LVIII Halftime Show mock player opened and visibly reports “NOW PLAYING,” but the new dialog did not receive focus. Focus remained on the underlying “Play Super Bowl LVIII halftime show” button, confirmed by focused-element metadata and Orca’s announcement. The dialog exposes “Close video” and “Pause video” controls in the accessibility tree, but focus was not moved into the modal, so their keyboard operation could not be reliably verified.

Keyboard + vision assisted

ap-m1-05-vision

MP4
Harness completedtrue
Task outcome (agent-reported)failed
steps
16
prompt_tokens
62414
prompt_cached_tokens
0
completion_tokens
3050
total_tokens
65464

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/nfl/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
required
flow_index
1
flow_title
Watch the halftime show from the most recent Super Bowl
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-M1-05 - New modal does not receive focus.

Watch the halftime show from the most recent Super Bowl

Source workflow steps: - Visit `/videos` from the Videos navigation item. - Select the “Halftime Show” category, then choose the featured “Super Bowl LVIII Halftime Show: Usher’s full performance” play control. - A full-screen mock player opens and visibly reports “NOW PLAYING”; its pause/resume and close controls work.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Opened the featured “Super Bowl LVIII Halftime Show: Usher’s full performance” player using only the keyboard and confirmed the visible “NOW PLAYING” state. AP-M1-05 was reproduced: focus remained on the background “Play Super Bowl LVIII halftime show” control instead of moving into the modal. Tab and Shift+Tab did not reach the visible Pause or Close controls, and Escape did not close the player. Therefore, pause/resume and close could not be operated by keyboard.

AP-N1-01: Navigation structure lacks usable anchors

A11y-tree navigation only

ap-n1-01-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
11
prompt_tokens
106091
prompt_cached_tokens
48384
completion_tokens
2367
total_tokens
108458

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/allrecipes/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
additional
flow_index
1
flow_title
Mock account login and logout.
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-N1-01 - Navigation structure lacks usable anchors.

Mock account login and logout.

Source workflow steps: Visit `/account`, enter any valid email and a password of at least four characters, then select Log in. The Alex Cook profile is shown. Select Log out to return to the login form.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Completed the mock account workflow using keyboard and Orca navigation only. Logged in with alex@example.com and an eight-character password, verified the Alex Cook profile, activated Log out, and verified the login form returned with the status “You are now logged out.” Diagnostic observation: Orca heading navigation produced no speech for the login/profile headings, so the controls had to be reached through the Tab order.

Keyboard + vision assisted

ap-n1-01-vision

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
12
prompt_tokens
62023
prompt_cached_tokens
0
completion_tokens
1883
total_tokens
63906

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/allrecipes/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
additional
flow_index
1
flow_title
Mock account login and logout.
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-N1-01 - Navigation structure lacks usable anchors.

Mock account login and logout.

Source workflow steps: Visit `/account`, enter any valid email and a password of at least four characters, then select Log in. The Alex Cook profile is shown. Select Log out to return to the login form.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Workflow completed using keyboard actions only. Logged in successfully, verified the Alex Cook profile, then activated Log out. The login form returned and the confirmation “You are now logged out” is visible.

AP-N2-01: Long page lacks a usable bypass

A11y-tree navigation only

ap-n2-01-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
32
prompt_tokens
319024
prompt_cached_tokens
152064
completion_tokens
7399
total_tokens
326423

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/bestbuy/replicate-01/g0-none/app/flows.txt
flow_section
required
flow_index
2
flow_title
View Bluetooth and wireless speakers on sale under $50.
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-N2-01 - Long page lacks a usable bypass.

View Bluetooth and wireless speakers on sale under $50.

Source workflow steps: Search “speaker.” In Filters, enable Bluetooth, Wireless, and On Sale, then enter 50 as Maximum price. The results update to the qualifying Insignia and JBL speakers, both below $50.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Completed the workflow using keyboard and Orca navigation only.

Verified:
- Search: “speaker”
- On Sale: checked
- Bluetooth: checked
- Wireless: checked
- Maximum price: 50
- Results: 2 items
  - Insignia™ - Mini Bluetooth Wireless Speaker - Black
  - JBL - Go Essential Portable Bluetooth Wireless Speaker

The individual displayed prices were not announced by Orca during the available reading attempts, but both products are present in the results with the Maximum price filter set to 50.

Keyboard + vision assisted

ap-n2-01-vision

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
29
prompt_tokens
190383
prompt_cached_tokens
0
completion_tokens
5170
total_tokens
195553

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/bestbuy/replicate-01/g0-none/app/flows.txt
flow_section
required
flow_index
2
flow_title
View Bluetooth and wireless speakers on sale under $50.
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-N2-01 - Long page lacks a usable bypass.

View Bluetooth and wireless speakers on sale under $50.

Source workflow steps: Search “speaker.” In Filters, enable Bluetooth, Wireless, and On Sale, then enter 50 as Maximum price. The results update to the qualifying Insignia and JBL speakers, both below $50.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Workflow completed using keyboard actions only. Applied On Sale, Bluetooth, and Wireless filters and set Maximum price to 50. Two qualifying results are visible: Insignia™ Mini Bluetooth Wireless Speaker — $24.99; JBL Go Essential Portable Bluetooth Wireless Speaker — $39.99.

AP-N2-02: Bypass target does not preserve main-content continuation

A11y-tree navigation only

ap-n2-02-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
13
prompt_tokens
88126
prompt_cached_tokens
55296
completion_tokens
1726
total_tokens
89852

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/adoptapet/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
additional
flow_index
7
flow_title
Complete a safe mock donation
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-N2-02 - Bypass target does not preserve main-content continuation.

Complete a safe mock donation

Source workflow steps: Open `/donate`, choose an amount, provide an email and name, and submit. A confirmation is shown and explicitly states that no charge was made.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Safe mock donation completed successfully. The confirmation says: “Your mock $25 gift confirmation is ready. No charge was made.”

Keyboard + vision assisted

ap-n2-02-vision

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
11
prompt_tokens
54588
prompt_cached_tokens
0
completion_tokens
1988
total_tokens
56576

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/adoptapet/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
additional
flow_index
7
flow_title
Complete a safe mock donation
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-N2-02 - Bypass target does not preserve main-content continuation.

Complete a safe mock donation

Source workflow steps: Open `/donate`, choose an amount, provide an email and name, and submit. A confirmation is shown and explicitly states that no charge was made.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Completed the safe mock donation using keyboard-only actions. Selected $50, entered test@example.com and Test Donor, and submitted. Confirmation shown: “Your mock $50 gift confirmation is ready. No charge was made.” Also observed AP-N2-02: activating “Skip to main content” left focus on the body instead of continuing at a main-content control.

AP-PS1-01: Focusable element is hidden from the accessibility tree

A11y-tree navigation only

ap-ps1-01-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
21
prompt_tokens
210028
prompt_cached_tokens
86016
completion_tokens
5780
total_tokens
215808

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/stubhub/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
required
flow_index
14
flow_title
Find out how to transfer NFL tickets to a friend.
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-PS1-01 - Focusable element is hidden from the accessibility tree.

Find out how to transfer NFL tickets to a friend.

Source workflow steps: Open Help, search “transfer NFL tickets,” and press Search, or choose Transfer NFL tickets under Popular topics. The article explains how to open an order, transfer selected tickets, confirm through the team partner, and have the friend accept them.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
To transfer NFL tickets to a friend:

1. Sign in to StubHub, open **Orders**, and select the NFL game.
2. Choose **Transfer tickets**, select the seats, and enter your friend’s name and email address.
3. Follow the link to the team’s ticket partner and confirm the transfer. StubHub will show **“Transfer sent”** in the order.
4. Your friend opens the email, creates or signs in to the team account, accepts the tickets, and adds them to their mobile wallet.

Transfers are free. The article recommends accepting the tickets before game day and using the team or venue app to scan in.

Keyboard + vision assisted

ap-ps1-01-vision

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
1
prompt_tokens
4784
prompt_cached_tokens
0
completion_tokens
333
total_tokens
5117

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/stubhub/replicate-01/g1-generic-a11y/app/flows.txt
flow_section
required
flow_index
14
flow_title
Find out how to transfer NFL tickets to a friend.
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-PS1-01 - Focusable element is hidden from the accessibility tree.

Find out how to transfer NFL tickets to a friend.

Source workflow steps: Open Help, search “transfer NFL tickets,” and press Search, or choose Transfer NFL tickets under Popular topics. The article explains how to open an order, transfer selected tickets, confirm through the team partner, and have the friend accept them.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
To transfer NFL tickets to a friend:
1. Sign in to StubHub, open **Orders**, and select the NFL game.
2. Choose **Transfer tickets**, select the seats, and enter your friend’s name and email address.
3. Follow the link to the team’s ticketing partner and confirm the transfer. StubHub will show **“Transfer sent”** in the order.
4. Your friend opens the invitation email, creates or signs in to the team’s account, accepts the tickets, and adds them to their mobile wallet.

The article also notes that transfers are free and tickets should be accepted before game day.

AP-W3-01: Supported activation has no relevant response

A11y-tree navigation only

ap-w3-01-a11y-nav

MP4
Harness completedtrue
Task outcome (agent-reported)completed
steps
48
prompt_tokens
489001
prompt_cached_tokens
248832
completion_tokens
12202
total_tokens
501203

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/airbnb/replicate-01/g2-taxonomy-informed/app/flows.txt
flow_section
required
flow_index
4
flow_title
Favorite a shared room in Belo Horizonte costing up to $100 for a weekend in May.
Exact task
Mode rule: use only keyboard and Orca navigation actions. Use the compact accessibility tree as a semantic map only; do not use mouse, click, coordinate, selector, direct accessibility-node activation, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-W3-01 - Supported activation has no relevant response.

Favorite a shared room in Belo Horizonte costing up to $100 for a weekend in May.

Source workflow steps: Visit `/stays`, select “Shared rooms,” type “Belo Horizonte,” choose a Friday-to-Sunday range in May, open “Filters,” set Maximum price to 100, choose “Shared room,” and select “Show stays.” On “Sunny room in creative townhouse,” select the heart button and choose “Favorites.” A screen-reader-perceivable “Saved to Favorites” confirmation appears.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Favorited “Sunny room in creative townhouse” in Belo Horizonte for the May 7–9, 2027 weekend. The applied filters were Shared rooms, maximum price $100, and Shared room type; the stay costs $64/night. Success is reflected by the pressed “Remove Sunny room in creative townhouse from wishlist” toggle and “Wishlists 1.” The exact transient “Saved to Favorites” confirmation was not directly captured by Orca after activation and waiting.

Keyboard + vision assisted

ap-w3-01-vision

MP4
Harness completedtrue
Task outcome (agent-reported)failed
steps
59
prompt_tokens
468386
prompt_cached_tokens
0
completion_tokens
14195
total_tokens
482581

flows.txt provenance

flow_file
${BLV_GENERATED_APPS_ROOT}/airbnb/replicate-01/g2-taxonomy-informed/app/flows.txt
flow_section
required
flow_index
4
flow_title
Favorite a shared room in Belo Horizonte costing up to $100 for a weekend in May.
Exact task
Mode rule: use only keyboard actions. Do not use mouse, click, coordinate, DOM-query, browser navigation buttons, extraction, or hidden implementation details.

Property under demo: AP-W3-01 - Supported activation has no relevant response.

Favorite a shared room in Belo Horizonte costing up to $100 for a weekend in May.

Source workflow steps: Visit `/stays`, select “Shared rooms,” type “Belo Horizonte,” choose a Friday-to-Sunday range in May, open “Filters,” set Maximum price to 100, choose “Shared room,” and select “Show stays.” On “Sunny room in creative townhouse,” select the heart button and choose “Favorites.” A screen-reader-perceivable “Saved to Favorites” confirmation appears.

Use only keyboard actions. When the source workflow says click, choose, open, select, or activate, perform the equivalent keyboard action with Tab, Shift+Tab, Enter, Space, arrow keys, typing, or waiting. Do not use mouse, click, coordinate, or hidden browser-inspection actions. Finish when the workflow is completed or the specified confirmation/result is visible.
Agent-reported final result
Partially completed using keyboard only. I set Belo Horizonte, selected the Friday–Sunday range May 7–9, 2027, selected Shared rooms, applied a maximum price of $100 plus the Shared room filter, and showed the matching stay “Sunny room in creative townhouse” at $64/night. The heart remains unselected, and I did not observe a “Saved to Favorites” confirmation before the step limit.