The eight hours I threw away

My bench hardware has a hub that captures two million ADC sample pairs a second and a tablet that records and displays them. The hub was the USB host and the tablet the device. I wanted that inverted, because when the tablet hosts, its own controller schedules the transfers and the ring overruns go away.

It took two sessions across two days. Nine hours thirty-four on the Saturday, seventy-two typed prompts. Eighteen hours forty-three starting three minutes later and running to Sunday lunchtime, ninety-five more. One hundred and sixty-seven prompts to move a USB role from one end of a cable to the other.

I deleted the first session’s work.

The prediction that did not hold

Nine hours in, on the Saturday afternoon, I typed this:

why role reversal broke everything in UI? why does the entire pipeline got disrupted?

the way I see it is: hub data structure remains same. the tablet reader structure changes, but not what it reads. rest of the downstream is same

That is a prediction, and it is the whole argument. The wire format does not change when you swap which end enumerates. Same 512-byte packets, same sixteen-byte header, same 124 pairs of raw ADC data. What changes is which side calls the transfer. The decode, the fold, the envelope, the recording path and the display should none of them know or care.

So when the UI broke, and the pipeline broke, and the recording path broke, I was not looking at a hard problem. I was looking at a wrong implementation. The blast radius contradicted the prediction, and the prediction was sound.

Three minutes later:

I would undo all the changes around role reversal, this seems very patchy

Nine minutes after that:

undo all the role reversal changes. and then give a very crisp list of things that we have to build for role reversal, discarding the path you have taken

No test had failed. The build was green. There was no error message and no stack trace. What there was, was a change that had touched far more than it should have been able to touch, and I could say how much more because I had said in advance what it should touch.

Why the mess happened

Two things went wrong at once, which is part of why it took nine hours to see.

The first was source control. My own words, twenty minutes before the undo:

the source control is completely broken, out of sync

I had a second session running against the same repository, working on the website. Commits were landing in overlapping places. By the time I asked “so which commits are related to role reversal, we have separate worktree>” the honest answer was that nobody could tell from the outside.

The second was worse. The work had gone into firmware/hub-usb, a bring-up tree from three weeks earlier that proves the hub can act as a USB device at 279 Mbit/s. Real code, working code, and not the product. The product is firmware/hub-usbhost, which carries the shipping pipeline: the staging DMA, the diagnostic frames, the pod receivers, the source control path.

Building the role reversal into the bring-up tree produced something that worked and was worthless. Every hour spent there was an hour spent on a branch of the tree that does not ship. When I restarted, my first instruction was:

but we build in the main, not into some obscure out of date bench code

What the second session did differently

Same feature, from zero, in a dedicated worktree, with a written spec and a phase order. The order was the point. Extract a transport-agnostic parser first, so both USB roles feed the same decoder. Then add the device arm behind a compile flag defaulting to off. Then flip the default, but only after the measurements are in.

The parser extraction is what made the rest small. One decoder, two transports, one attach path. After that the role reversal is a #define and a couple of hundred lines of endpoint handling, which is what Saturday’s prediction said it should have been.

Verification was a ten-minute soak at the production rate: 1,201,010,432 sample pairs, 2.0000 Mpair/s against 2.0000 nominal, 4.959 GB moved, zero dropped packets, zero sequence gaps, zero overruns. Sixty-three commits landed on the firmware and app trees across the two days, each push gated on a suite that now runs 588 tests.

The counters that made it legible

That tile reads, verbatim:

   0          0
 gaps      overrun
  0k         —
discard   overwrite

Three of those are counters the hub reports. The fourth I derived on the tablet by subtracting what the hub had delivered from what its ADC had landed in memory, and it is a dash now because I could not make it honest.

It read 2,562k on a link with nothing wrong with it. I spent a chunk of Sunday proposing mechanisms for that, seven of them, each killed by a measurement, before reading the two lines that increment the underlying counters. One counts every lap of the ADC’s ring buffer since the hub powered on, including while USB is unplugged. The other counts whole packets that reached the page. They count different populations. Subtracting one from the other gives you the capture that happened while nothing was listening, which is a constant rather than a leak.

I tried three ways to subtract that constant back out. All three failed on hardware. So the cell renders a dash, because the honest reading of a quantity you cannot measure from what the wire carries is that you cannot measure it.

What I would tell myself

Say what a change should touch, before you make it, in a sentence a stranger could check. Then the gap between that sentence and what the diff actually touched is a signal you can act on, and it fires earlier than a failing test, because a wrong implementation that is internally consistent passes every test you have.

The nine hours were not entirely wasted. They produced the prediction that let me throw them away. But I would rather have written that prediction on the first prompt than in the tenth hour, and nothing about writing it down early requires an AI, a bench, or two thousand lines of firmware.

The duller lesson cost the same: know which tree ships. A working branch that nobody runs is not progress, and from the inside it looks exactly like progress.