We Added Shot Noise to Our Own Benchmark and One Optimizer Fell Off a Cliff

How the experiment was built

The Hamiltonians are not new. Each published QEncode entry stores its complete Pauli decomposition, so we reconstructed the operators directly from the artifacts — the exact ground state of the reconstructed Hamiltonian agrees with the stored value to 2.7e-15 Hartree. Anyone can do the same from the public repository.

On top of each Hamiltonian we ran a hardware-efficient ansatz started from the Hartree-Fock determinant, matching the suite’s own construction, and optimized it with three methods across three sampling regimes and ten random seeds each. Every run was single-threaded, with the sampling RNG seeded, so every number below is reproducible.

One methodological point that mattered more than anything else: we calibrated the optimizers at zero noise first. Our first attempt had Adam at 292 mHartree against COBYLA’s 0.73 with no noise at all — an unequal iteration budget, not a finding. Had we skipped that check we would have published “gradient methods collapse under shot noise” backwards. In the results below all three methods land within 0.06 mHartree of each other on water at exact evaluation, so every later difference is sampling and nothing else.

Water: COBYLA falls apart, the others do not

At 1,000 shots COBYLA failed in ten out of ten seeds, landing above 100 mHartree every time. Adam failed in none. This is the clean comparison in the whole study: identical Hamiltonian, identical ansatz, identical starting distribution, and all three methods provably converged before noise was introduced.

Lithium hydride: the cliff

Then the same experiment on a larger Hamiltonian produced something we did not predict.

At 10,000 shots the two are indistinguishable. Drop to 1,000 and water barely moves while lithium hydride collapses — ten seeds out of ten above 100 mHartree. Adam over the same range went from 1.07 to 4.94 mHartree, with no failures.

The mechanism is not exotic. L-BFGS-B builds a curvature model from gradients; with parameter-shift evaluation each gradient component is itself a sampled quantity. Estimating an energy means estimating a sum over Pauli terms, so at a fixed shot budget the sampling error grows with the number of terms. Past some point the gradients stop carrying usable curvature information and a quasi-Newton method has nothing left to work with. Adam does not build a curvature model, so it degrades smoothly instead of falling over.

The practical consequence is a rule of thumb worth stating plainly: the shot budget a gradient-based optimizer needs is set by the size of your Hamiltonian, not by how correlated your molecule is. A method validated on a small active space can fail on a larger one at the same shot count, with no warning.

Why the literature disagrees with itself

Before drawing conclusions we checked what was already published, and found the field split. One benchmark of optimizers for quantum chemistry — covering H₂, LiH, BeH₂, H₂O and HF, essentially our molecules — reports that “in noisy quantum circuit conditions, SPSA, POWELL, and COBYLA are among the best-performing optimizers”. Other work reports the opposite, that COBYLA and Nelder-Mead are among the most heavily damaged by noise.

Our data reconciles them, and the resolution is mundane. First, noise type: gate and decoherence noise bias a smooth objective, which a trust-region method tolerates; finite sampling makes the objective stochastic, which it does not. Second, what the comparison is against: COBYLA looks strong beside deterministic quasi-Newton methods that break on noisy gradients, and weak beside a stochastic optimizer. Third, shot budget versus Hamiltonian size, as above. Two careful studies can rank the same optimizer oppositely and both be right.

We want to be clear that none of this is a novel discovery. Optimizer robustness under noise is well-studied. What we think is useful here is the controlled contrast — one Hamiltonian family, matched noiseless baselines, seeded sampling, every run published — which turns a disagreement between papers into a statement about when each answer applies.

What this means for our own numbers

It reframes them, and not flatteringly. Our certified entries reach accuracies well below what sampling can resolve at realistic budgets. One LiH entry certifies at 0.096 mHartree; the sampling spread on that same circuit at 100,000 shots per energy evaluation is about 2.5 mHartree — roughly 26 times larger. Resolving the certified figure would take on the order of 10⁷ shots per evaluation, and a VQE run needs thousands of evaluations.

That is not an argument against noiseless benchmarking. Separating algorithmic accuracy from sampling is exactly why the suite is defined the way it is, and you cannot attribute a failure to the ansatz if shot noise is free to move the answer. But a certified gap is a statement about an algorithm, not a prediction about a device, and we would rather say so than let the number be read as more than it is.

What we got wrong

Three predictions, recorded before measuring, and the data overturned all of them.

• We predicted both optimizer families would degrade under sampling, since parameter-shift gradients are sampled too. Adam barely degrades.
• We predicted L-BFGS-B would collapse like COBYLA, reconciling the literature neatly. It collapses on one molecule and is fine on the other.
• At 88% of the runs complete we reported a clean three-way ordering on water. At full statistics it disappeared — L-BFGS-B came in at 9.1 mHartree, not the 65.6 the partial data showed, statistically tied with Adam.

The last one is the reason to run every seed before publishing. A partial result that looks tidy is the most dangerous kind.

Limits

Two molecules, one ansatz family, three optimizers, ten seeds. Shot noise only — no gate error, no readout error, no error mitigation. COBYLA on LiH fails even at exact evaluation (61 mHartree), so its degradation there is not cleanly attributable to sampling, and we have not used it for any claim. The threshold we describe is bracketed between 1,000 and 10,000 shots on two Hamiltonians; we have not located it precisely, and we do not know its functional form.