Toss Nebula device farm
Nothing transferred. What it gave me instead was a ruler — and held against my own code, it found a crack.
Zero techniques, zero adoptions. The one thing I adopted in the first pass was rejected in the second — the measurement behind it was correct, and that call appears zero times in my pipeline. A game client paints the whole screen into a single native surface, so there is no accessibility tree to read, and that fact was already written in my own project docs. The investigation still wasn’t wasted: using one of this system’s principles as a ruler against my own code turned up two gateways where there should have been one.
What they built
It started by consolidating the small farms each team ran on a Mac mini with half a dozen phones plugged in. Five Mac minis, fifteen devices and one developer at the start; past a hundred devices within a year.
This is all a user sees.
POST /device/occupy { platform: "android", tags: ["smoke"] }
POST /actions/click { x: 540, y: 1200 }“Grab me a device → press (540,1200) on it.” You never need to know which Mac mini it’s attached to, or how the driver is configured.
They rebuilt from the device layer instead of extending the existing tool
That’s the biggest decision in the piece. Scaling up, they kept hitting structural limits in the existing automation framework, and judged owning it better than patching it.
| What was removed | Why |
|---|---|
| The session concept | 15–40 seconds to start a session every time, session failures climbing with scale, management cost proportional to device count |
| Desktop-oriented mirroring | Existing tools assume “device screen → desktop app,” which doesn’t fit fan-out to browsers through a server |
| Capture that monopolises USB | Monopolise it and you can’t drive the device at the same time. The alternative was a 10–15 fps slideshow |
In place of sessions they put an always-running controller with stateless HTTP calls on top. And the interface is defined by one spec, from which clients are generated.
The side effect of that ownership is the real value. Their own Korean input method (characters don’t break), instant installs of internal builds, security policy enforced inside the driver spec itself — owning the spec means never waiting for someone else’s tool to support you.
What broke — 13.5× isn’t a like-for-like comparison
The headline: an element click at 52 ms against 702 ms — 13.5×.
But the author attached his own caveat: roughly 80% of that gap is the existing tool’s wait-for-idle, and turning that off narrows it to 2–3×. So it’s a comparison between a safety mechanism on and off. Stating that is honest; quoting “13.5×” out of the table isn’t. Their choice is reasonable — driving a device while watching it live wants speed. But it’s a trade in philosophy, not a free win.
The bigger gap is elsewhere: there’s a p50 and no p99. What kills test automation is the tail, not the median. Session and command failure rates, device uptime, reduction in flakiness — not a single stability figure appears anywhere in the article.
The rest, for the record. The effectiveness evidence is two testimonials with no measurement conditions or sample. The nearest open-source prior art is absent from the comparison — “why not that, instead of building our own” goes unanswered. The iOS capture path is tied to one OS layer, producing a structural cost of not being able to unify hosts, and that isn’t written up as a cost. And the driver is closed, so no third party can verify the table.
I measured my own environment — and the target was wrong
One real device, read-only operations only (anything that drives the device changes its state).
The UI dump ran 2,350 ms, ten times slower than theirs. So the first pass adopted the in-device resident controller as its one harvest.
The second pass rejected it. Checking the code exhaustively, that call appears zero times in my pipeline.
The reason was already in my project documentation. A game client paints the entire screen into a single native surface — there is no accessibility tree to begin with. Which is why my screen driving is a screenshot → vision → coordinate-tap loop. Attach that tool and an empty tree comes back. The thing that would get ten times faster does not exist.
The 2,350 ms measurement was right and the target was wrong. Before benchmarking anything, check whether that call actually happens in my run.
The same measurement killed two more hypotheses. “Switch to cable and it gets faster” — transfer is 5% of the total, so the ceiling is 5%. “Keep a process warm and reuse it” — the round trip is 73 ms, so the ceiling is 4%. Their gain comes from inside the device, and reading it as a host optimisation wastes the work.
What stayed wasn’t their technique — it was my own discipline
I took the point where the article says “every tool was built on the one public API” and used it as a ruler against my own code.
Places that bypass the gateway and call the command directly: 14 occurrences across 7 files. Roughly half against the files that go through the wrapper.
And something worse turned up. The bypassing files weren’t the problem — there were two gateways.
| Locking | Error classification | What real runs use | |
|---|---|---|---|
| The official gateway | yes | yes | ✗ |
| The other one | no | no | ✓ |
Not one of the official gateway’s defences was wired into the path a real run takes. When I later attached instrumentation, that fact forced me to attach it to the undefended side.
Verdict
Zero techniques, zero adoptions. What this investigation produced is one finding about my own code.
Sometimes that’s the honest result of a benchmark. Even when there’s nothing to copy from someone else’s system, holding its principles up as a ruler against yours shows you where yours has split.
The rejection carries a resume condition too — when the target becomes an ordinary app, or the build starts exposing accessibility nodes.