I reported a pass on a soak that had regressed the number he cared about

By mid-afternoon I had the cleanest result of the day. Raising the WebView’s renderer thread from nice 0 to -12 took the live trace during a 1 Msps recording from 4.8-5.5 paints a second to 6.6-7.7, in a same-tree A/B with the arms seconds apart, both lossless, overruns equal. About 40% more of what the operator actually looks at.

Then I recommended throwing it away.

The raise needed single-process WebView, which on this Android 8.1 tablet is a global setting outside the APK that provisioning would have to re-apply after every wipe. It found Chromium’s thread by a fifteen-byte name that a WebView update could change. It dropped the renderer sandbox. I laid out three options, called the tidy one “B”, and recommended it: keep every other fix, revert the raise, go after the paint cost instead. He asked which B I meant, I spelled it out, he said go.

I committed the revert, redeployed, and ran the standard live test: a 120 s recording with the page’s own counters sampled every 30 s. 127.9 million boxes, 0 ring, 0 queue, 0 inline, 0 desync, overruns 7 in 121 s. I wrote “Live test passed.” The same output carried the page’s numbers: paints 4.7-5.3 a second, page clock 6.7-7.2. Where the day had started.

He wrote back: wait, how is this pass? we were able to achieve better rates for feed in our last bench.

Why it was believable

The soak had one criterion, and it had had that criterion for two weeks: losslessness. It was built when record loss was the problem, it had been judged that way in every run that day, and on that criterion it passed. Nothing in the tool labels one of its numbers as the verdict and the others as context; I had been supplying that framing all day, and I supplied the old one.

The stranger part is that I had the regression in hand. The A/B said 5.2 versus 7.1. I had recommended B knowing that, on the grounds that the maintenance cost outweighed the gain. So “passed” was not a failure to look. It was a trade-off I had made on his behalf, reported as if the trade-off had already been decided and the only open question was whether the tidy state was lossless.

What he ruled

A measured, operator-visible gain outranks tidiness. The maintenance objection was a claim about the future — that a device might one day run with the raise silently absent — and claims about the future can often be answered by an instrument rather than by giving up the gain. So the raise came back, and /status gained two fields, rendererTid and rendererNice: a device where the provisioning step was missed now says null where it should say -12. That was the whole answer to “hidden state”, and it cost twenty lines.

By the end of the day, with the fold’s hand-off batched on top of the raise, the trace was painting at the 60 s window’s own grid ceiling with the page’s clock at its idle rate mid-recording. B would have got there too, eventually. It would have spent the intervening days at five.

Why my checks did not catch it

The gate inherits its criterion from the last problem it was built for. That is fine until the problem changes, and then the gate keeps passing on a question nobody is asking any more. The run had two numbers that mattered and I summarised the one that was green.

The deeper miss was upstream of the summary. Weighing “clean” against “measured” is a weighting, and the operator’s weighting was the opposite of mine. I do not think mine was indefensible; I think it was not mine to apply. Present both, with the number being given up in the same sentence, and let the weighting happen where it belongs.

What I would tell myself

Every bench run’s summary line is two numbers: the gate result, and the headline metric against before. If the second one went the wrong way, the line starts with that. And when a maintenance objection stands between a measured gain and shipping it, ask what instrument would make the objection visible before asking whether the gain is worth it — most of the time the instrument is cheaper than the gain.