I did not touch the threshold
One run, 126 cases, one failure. I knew how to make it pass, and didn't.
126 cases, start to finish. 937 seconds. One failure.
I knew how to make that one pass. Widen the verdict window from 30 seconds to 60. One line.
Why there are five verdicts
With only PASS and FAIL, automation lies. Cases the machine isn’t sure about have nowhere to go, so they get pushed into one of the two — and the one they get pushed into is usually PASS.
So there are five. PASS and FAIL only when a log anchor lets the machine be certain. EVIDENCE when the screen was captured but nothing can be concluded. SKIP when the case never ran. NA when it’s out of scope.
| Verdict | Count |
|---|---|
| EVIDENCE | 66 |
| SKIP | 39 |
| PASS | 16 |
| NA | 4 |
| FAIL | 1 |
The largest block is EVIDENCE, at 66. That means the machine decided on its own for fewer than half the cases — and not hiding that number is the whole point of splitting the verdicts up.
The one failure
Two consecutive cases were judging the same action against the same log anchor. The first waited 30 seconds; the anchor arrived after that and was caught inside the second case’s window.
The event fired normally. The first window simply missed it. Four other cases with the same title all passed in this run, and in the previous run the anchor arrived inside 30 seconds and this case passed too.
Which is to say the result is decided by timing.
Widen the window and this run goes green. The cause stays exactly where it was, and the next run wobbles in the same place.
The structural problem is a duplicated anchor. The fix is to give the first case an anchor of its own, or merge the two — and either way the expected values have to be confirmed first. This run closed carrying one failure.
The diagnosis I got wrong
Nine of the 39 skips reported as “tap target unmapped.” I wrote that filling in the coordinates would recover them. I called it the cheapest coverage I could buy.
Wrong. Filling them in recovers nothing.
Dumping all 7,455 widgets showed the targets existed and were named. What blocked them wasn’t coordinates — it was reach. One ancestor container was not visible, so the entire left HUD stayed folded, and it never opened once during the run.
| Of 72 in-game dumps | Appeared |
|---|---|
| Right HUD | 72 |
| Left HUD | 0 |
Adding coordinates only changes the skip reason from “unmapped” to “action failed.” Growing coverage needs a route to that screen, not a mapping.
One string dragged the entire diagnosis in the wrong direction. A reason string isn’t log decoration — it’s the interface a person reads to decide what to do next.
Where the time went
Of 937 seconds, 302 (32.3%) were bridge round trips. Of those, dumping the widget tree alone accounted for 186 seconds — 19.8% of the whole run.
That’s what happens when every judgment pulls the entire tree. A call that asks for only the widgets in question would cut it. By how much, I don’t know until I measure before and after against the same set of cases.
Until I saw these numbers I assumed shader compilation was the bottleneck.
Verdict
The one failure was not a product defect. And I still didn’t turn it green.
Being able to write both of those sentences at once is why the verdicts are split five ways and why the reason strings get written down.