"Probably right" is not a verdict

There are two ways to answer the same question. “It’s probably right” and “it’s right.”

I saw in somebody else’s talk that the difference between them isn’t effort.

One query beats hundreds of pulls

The job is confirming that a gacha rate matches the design. Without access, that meant pulling hundreds of times and estimating the rate from the sample. Once read access existed, it meant reading the source in five seconds with one query.

It didn’t get faster. The grade of the evidence changed — a sample moved to a direct read, which on this table is a jump of several rows. And the verdict changes with it: one form can only ever reach probably, the other settles the question.

And what made that difference wasn’t a tool, it was access.

I graded them and held my own verdicts against it

Grade	What	How far it lets you speak
1	read the source directly	right / wrong
2	deterministic code checks it — size, count, hash	present / absent
3	a log anchor	happened / didn’t
4	screen captures	a person has to look
5	sample statistics	probably
6	somebody’s self-report · a document’s prose	not evidence

The table isn’t novel. What earned its keep was counting what grade the evidence I actually used was.

The place I got it wrong most was documentation

I summarised an MMO server repository from the README alone. Opening the code showed two things wrong, pointing in opposite directions — I claimed a feature that doesn’t exist and denied a privilege that does.

What was wrong was not the document but the grade I handed it. That file stated plainly what it was and how far it had been built. All I did was seat it in grade 1 — and ungraded, grade 6 takes that seat without resistance.

There was a worse one. Benchmarking an action game, I remembered that the reward for a perfect dodge was slow motion, and on the strength of that memory I was about to build a prerequisite. Cross-checking the sources showed the actual reward is a resource gain and slow motion was confirmed nowhere.

Memory isn’t on the table at all.

I treated a self-report as evidence twice

One agent browser leads with “first place on agent benchmarks”, and that number was posted to their own repository under their own scoring setup. No third party has reproduced it.

Another framework has a policy of “we don’t change it without evidence” — and that evidence lives in a separate repository, so it can be neither verified nor refuted.

Both claims may well be true. They are still grade 6, and you don’t decide an adoption on grade 6.

Evidence that isn’t there looks like evidence unless you count it

One talk runs 31 minutes, and with no captions I reconstructed it from slides alone. Only half of it was analysable, and the missing half was the answers. What survived is a list of questions, and a list of questions is not a finding.

Write only the summary without recording the gap, and later, when that page gets cited for something, the hole is invisible. Which is why that piece opens with “this piece has to start by admitting its evidence is thin.”

What I didn’t do

Nothing enforces the grades. Right now I hold to them by hand while writing. My automated pipelines do leave grounds behind for every ruling, but they never record what grade those grounds are. Leaving a record and grading it are different jobs.

And ⚠ there are almost certainly places where I still use grade 5 as if it were grade 1. This piece records the three I found; it isn’t a full audit.

Verdict

Skip the grading and you end up making the strongest verdicts on the weakest evidence. And the reason that’s comfortable is obvious — grade 6 is the easiest to get. A README takes thirty seconds, somebody else’s number can just be copied down, and pulling up my own memory costs nothing at all.

Grade 1 usually requires being granted access. Which means the real problem in this category isn’t verdict technique but what I am allowed to open. The gacha case was exactly that story.