I calibrated an instrument, checked it against itself, and reported 8.5 mV. The real error was 521 mV.

A programmable voltage source needs a table mapping DAC codes to output volts. The linear law it shipped with was off by up to 465 mV, because the analog stage is not linear — so I swept every DAC code, recorded what came out, and built a 113-knot lookup table from the result.

Then I verified it. Set 1 V, measure. Set 5 V, measure. Eleven points across the range:

set_mv  code  meas_mv   err
 1000  3699    997.8    -2.2
 5000  2747   5003.4    +3.4
13000   848  12998.7    -1.3
15000   372  14993.5    -6.5

Max error 8.5 mV, mean 3.3 mV. I reported that as the table’s accuracy and committed it.

Later the operator moved a precision DVM onto the output terminals — a separate instrument, on the actual binding posts, resolving in tenths of a microvolt. I re-ran the same setpoints against it:

OLD table vs precision DVM: max 521.0 mV, mean 41.4

Sixty times worse than I had measured, on the same board, at the same setpoints, minutes apart.

Why the first number was believable

It was not a fluke or a bad point. All eleven setpoints agreed. The errors were small, scattered in sign, and centred near zero — exactly the signature of a good fit. If you had shown me those numbers cold I would have called the calibration excellent, and I would have been reading them correctly. The measurements were honest.

The problem was upstream of the measurement. I had built the table from the board’s own voltage sense — a 12-bit ADC behind a divider, giving a resolution of about 10.5 mV at the output. Then I verified the table using that same ADC.

Why the check could not fail

Any error the ADC makes, it makes twice: once when the table records what a code produced, and once when the check asks what that code produces now. Both readings carry the same quantization, the same divider tolerance, the same reference error. Subtract them and all of it cancels.

So the check was not weak. It was structurally incapable of detecting the error class I most needed to find. It could catch a code applied wrongly, a transcription slip, a knot in the wrong slot. It could not catch the table inheriting the ADC’s own error, because from inside that loop, the inherited error is zero.

There was a second signal I missed for the same reason. Earlier I had noticed adjacent setpoints jumping in ~10 mV steps and started reasoning about whether to dither the DAC for finer control. But the DAC’s step at the output is 4.21 mV and the ADC’s is 10.51 mV. The source was already finer than the instrument watching it. I was about to improve the half of the loop that was not the problem — and I would have measured the improvement with the instrument that could not see it.

What the second instrument changed

The rebuilt table, swept against the precision DVM, verifies at max 5.98 mV and mean 1.69 mV. That number means something the first one did not: it was produced by a device that shares no front end with the thing being calibrated.

The static noise on that DVM at a fixed code is 0.395 mV. The board’s own ADC quantizes at 10.51 mV. The entire error I had been unable to see lived in the gap between those two figures.

What I would tell myself

Agreement between two readings of one instrument proves the instrument is self-consistent. It says nothing about whether it is right. That is obvious stated plainly, and it was completely invisible to me while I was inside it, because every individual step was sound — the sweep was careful, the fit was monotonic, the verification was real code producing real numbers that really did agree.

The rule I would want in front of me: before quoting an accuracy figure, name the instrument that produced it and the instrument that checked it. If they are the same, you have measured repeatability, not accuracy — and you should say “self-consistent, not independently confirmed” instead of quoting a number.

The uncomfortable part is that I had a validation, it passed, and it was worse than useless — it converted an unknown error into a confident wrong number, and I committed on the strength of it.