Asleep's QA automation

Exactly one technique transferred. Pulling eleven design decisions from eighteen months of writing by the sole QA engineer at a sleep-tech company and holding them against mine, ten were places I had already reached. The one remaining is this page’s harvest — but the real return of this investigation wasn’t that one thing. It was that under the same constraints, someone else independently reached the same decisions. And getting to that verdict, my own conclusion was overturned twice.

Who built it

The sole QA engineer at a company making a sleep-measurement app. Eight posts between January 2025 and August 2026, at very different levels — one on career motivation, one on post-deploy monitoring, one on code standards for large-scale automation, one debugging a memory collapse during a 24-hour run, one on building an on-premise device farm, and a retrospective on their own QA platform.

It matters that they tried to stay on the cloud first. A cloud device farm’s beta failed during an audio-injection trial, and the vendor’s proposed fix was downgrading to a framework version that had been end-of-life for five years. They pushed back — that version predates the shift from synchronous to asynchronous architecture — and it eventually worked on the current release. Even so, the way audio injection works turned out to be wrong for validating this product, and mobile left the cloud.

And the author’s conclusion is honest. Having argued a cloud vendor down and won, it ends on “for most people, cloud is the right answer.” On-premise is a good direction only when a special requirement can’t be met in the cloud and can be designed well as hub-and-node. Writing that declines to generalise its own choice is rare.

The core is simultaneous control — not parallel execution

Conflate the two and you misread the whole series. They’re completely different axes, and the one the author calls the core is the second.

	Parallel execution	Simultaneous control
What N devices do	different tests each	the same action, at once
Purpose	throughput, overnight sweeps	compare behaviour across models by eye
Hardest part	session collisions, dirty device state	pressing the same button
Solution	hub scans free nodes and queues	target the real on-screen element, not coordinates

The author defines simultaneous control as “not a convenience feature but the feature that changes how much verification one QA engineer can carry.” Checking twelve configurations one device at a time means repeating every action once per device; simultaneous control deletes the repetition — two devices or ten take about the same time.

Two conditions had to hold.

• Accuracy — ratio-computed coordinates drift with each model’s margins and status-bar height, missing by over 100px on the same screen. So targeting moved to finding the actual element
• Latency — if one device lags, the whole set feels out of step. So they bypassed the standard framework for a direct device connection, and measured 14 ms for a screen tap and 4–7 ms for a keystroke over wireless

And the best-formed number in the whole series happens to sit here. It states its conditions (“measured over wireless”) and separates the two operations.

What broke — there is no defect-detection data

One gap runs through all eight posts.

A device farm, a platform, a natural-language verification engine, 900 of their own tests — and nowhere is there a number for “we caught N defects that manual testing would have missed.”

Everything is evaluated on whether it runs, and nothing on whether it finds. That is precisely where automation most commonly fails.

The rest, on the record. “Recovers within 45 seconds” can’t be verified — the detection mechanism isn’t described, so it’s likely a configured timeout rather than a measured recovery latency. In which case 45 seconds is a parameter, not an achievement. “900 of our own tests” is a vanity metric — no coverage, no defect count, and since the same person wrote both the platform and its tests, it’s a self-referential oracle that replicates its own blind spots.

Zero evidence on the most fragile component also stands out. Unattended natural-language scenario verification is the most brittle part of the platform, and there isn’t a line on accuracy, flakiness, or behaviour under ambiguity.

And the memory-collapse post is half an expansion of a wrong hypothesis. The three-layer timeout material is useful knowledge but wasn’t the cause; the actual fix was one line — write recording segments straight to disk instead of accumulating them in the heap. And raising every timeout to 30 hours switched off the hang detector. Justified for a 24-hour test, but as a standing default it removes any way to distinguish a stalled session from a long one.

One thing came across

The one that came across:

Count the devices that failed element lookup and got tapped by coordinate instead — and show that count immediately.

I have the coordinate fallback. I don’t have the counter. So a tap that hit its target precisely and a tap that missed the element and got approximated by coordinates are recorded as the same PASS.

There are two kinds of PASS and the report records one. The fallback path is exactly where the next build breaks quietly, and without a count you won’t see it growing.

And I corrected myself twice

First. I recorded silent-connection detection as a new harvest. Wrong — opening my own code, it was already there. A separate axis tracking screen change, wired to a staged recovery. I had mistaken a written operating rule for an implemented one, which is the identical error I made in Prime Agent.

Second. So I tried to close at “zero new adoptions,” and that was wrong too. The section the author himself calls the core hadn’t made it into my initial extract, and the one adoption above was sitting in it.

A thin extract changes the conclusion wholesale.

Verdict

One technique transferred. The real value of this investigation was elsewhere.

Under the same constraints — one QA engineer, real devices, long sessions, unattended regression — someone else independently reached the same decisions. On-premise, fail-closed on live by default, unattended regression, element-based targeting. Those were conclusions I’d arrived at alone, which meant a sample size of one. Now it’s two.

Few techniques transferring and the investigation being wasted are two different statements.