All four times, the numbers were right

Measurement usually goes wrong somewhere other than the arithmetic.

Four of my verdicts reversed over the last few months, and all four times the arithmetic was right. The scripts counted correctly and the logs recorded correctly. What was wrong was what I measured against.

Without a control, measurement confirms what you already believe

I picked up a criterion from someone else’s rule set — match the form to the failure — and ran it over my own rule documents. Prohibitions outnumbered recipes by 12.6 to 28.6 times, with recipes at effectively 0%. I wrote “my corpus is abnormal” and adopted it as the top item.

Their own rule already said what to do here.

Always include a no-guidance control. If the control doesn’t exhibit the failure, there is nothing to fix — stop, and don’t author the guidance.

My control was the reference implementation. I ran the same script over it. The reference implementation as a whole came out at 26.6×, and the document making the argument sat at 18.5×. Mine were less skewed.

The numbers were right from the start. There was no baseline. And with no baseline, I reached a conclusion anyway.

Ask whether there is anything to improve before asking how much

Evaluating a context-compression layer, I set out to measure what adopting it would gain. First I measured my own side. Cache reuse 92.5–96.3%, uncached input 1.8–2.0 tokens per call.

There was nothing left for compression to compress. The tool wasn’t bad — there was no problem on my side.

Ask “how much does this improve things” first and it always looks like an improvement. The order has to flip — is there anything to improve.

Totals hide the cause

One run of 126 cases took 937 seconds. I believed shader compilation was the bottleneck.

Instrumenting each segment gave a different picture.

Segment	Time	Share
Bridge round trips	302s	32.3%
└ one tree dump inside that	186s	19.8%

It fetches the whole tree every time it makes a call. Shaders weren’t even near the top.

A total conceals the cause. Until I measured segment by segment my guess had nothing behind it — and I didn’t know it had nothing behind it.

Counting inventory is not measuring

Holding my own operation against someone else’s four-rung ladder, I wrote equivalent into four cells. The baseline behind all four was whether it was installed.

The arithmetic was right here too. Measured with an instrument that asks “is this installed”, those four cells genuinely are equivalent, and I applied that instrument correctly. What was wrong was the instrument — I judged “it runs” with a tool that measures “it exists”.

Pressing the last cell for real returned no items, no files, no observer hooks. Possession is not operation. And unless the instrument changes, the other three cells stay exactly as they are — never pressed, still written down as equivalent. A value read off the wrong instrument is wrong in the same direction whether you read four cells or forty.

Which is why unmeasured numbers get a label

That is how the convention came about. Anything I can’t measure is published as an estimate with an interval and an [estimated] label, and to promote it to measured, 10% is held back untouched.

One recent adoption sits in exactly that state. Declaring a quality bar in one word inside a prompt — the output does look better, but whether that came from the word or from the other conditions was never separated. With a sample of one there is no interval to give.

So it is adopted, labelled [estimated], and the promotion path is written next to it — build 10% of future reports without that sentence and compare.

What I didn’t do

I have never once run the holdout. Writing the method down and having measured it are different things. One of the five sections above is still [estimated], and it went into this piece in that state.

And I never finished applying the control rule to myself. One of four “equivalent” cells has been pressed. The other three are still ledger entries as I write this.

Verdict

The four misjudgements share something. In every one, my own numbers came out well, and there was nothing to compare them against.

A control, the baseline before it, segment decomposition, and pressing the button. Four names for one job — making a place where an answer different from the one I already believe can appear. Without that place, measurement becomes a confirmation step, and confirmation steps have never once been wrong.