We Said Every Result Was Reproducible. Then We Tried to Check.

A claim we had never tested

Every QEncode entry records everything needed to rebuild it: molecule, basis, active space, mapping, ansatz, optimiser, seed, iteration budget, package versions, and the commit that produced it. verify_entry.py re-runs an entry from that record and compares the energy. That tool is the whole reproducibility claim, made executable.

We had run it on individual entries many times. We had never run it on all of them. So we did.

Bug one: it could not run 29 of 54 entries

Entries record the ansatz in the pipeline’s internal vocabulary. The command line takes a different one. 29 entries store ansatz_type: "hea"; the flag accepts only hardware_efficient. Every one of those verifications died at the first call:

A sibling case, uccsd_tapered → uccsd, was already handled by a string replacement. hea was simply never mapped.

Bug two: three entries re-ran at the wrong budget

With that fixed, three entries still failed, by up to 221 mHa. The cause split the database perfectly on a single field:

The verifier never passed the recorded iteration cap. Entries needing more than 500 iterations were silently re-run at 500 and landed somewhere else. One line. All three then reproduced — the worst of them, N₂, in 2.1 hours.

Bug three: it only worked on our machine

The pipeline refuses to write an entry from a dirty git tree or a drifted package set. That is correct. But the verifier had no way to override it, so verification was impossible for anyone whose environment was not byte-identical to ours — which is everyone checking our work from outside, and the only audience the tool has.

Three bugs. All three in the checking. None in the results. After the fixes, all 54 entries reproduce — including H₁₀ at 20 qubits, which took two hours, and H₆, which took 2.1. Eight hours of compute to check the database once.

Then we put it in CI, and CI failed

A check nobody runs is not a check, so we added it to continuous integration: a six-entry subset on every push, the rest weekly. The first weekly run failed every shard.

Not a new bug. We had asked CI to confirm that a regenerated energy matched the published one to 10⁻⁶ Ha, on a different machine. The numerics do not support that, and we already knew why.

A gradient-free optimiser like COBYLA picks its next step by comparing two nearly equal energies. A difference in the thirteenth decimal can flip that comparison, change the step, and land the run in a different local minimum. We had measured exactly this between two simulator backends on the same machine: agreement to 2.6 × 10⁻¹³ Ha on a single energy evaluation, and 11 mHa apart after COBYLA. A different machine is a bigger perturbation than a different backend.

So we measured what actually holds. Re-running 40 entries on a drifted environment:

17 of 40 exceeded the strict tolerance while still being perfectly valid results. That gap — between “the energy is identical” and “the entry is still correct” — is the whole point. Only the second is a property that travels between machines.

What reproducible has to mean

We wrote the definition down rather than letting CI imply a stronger one than the data support. Three claims, in decreasing strength:

1. The procedure is identical and fully declared. Same ansatz, optimiser, iteration budget, seed, active space, mapping. All recorded.
2. The outcome still satisfies the certification criterion. The regenerated error is still under the threshold. This is what should hold on any machine.
3. The energy matches bit for bit. Claimed only on the reference pinned environment.

Anything stronger than (2) across machines is aspirational, and currently false for COBYLA-style methods. CI now checks (2). The pinned reference environment checks (3).

The finding that outlasts the bugs

With CI asking the right question, two entries still failed. Not a bug — a property of those entries:

The certification threshold is 10 mHa. Both were certified. Both stop being certified when rebuilt somewhere else.

An entry certified close to the threshold is not robustly certified. Our leaderboard had been showing an entry at 9.6 mHa and one at 0.001 mHa identically — both simply “certified”. They are not the same thing. One survives being re-run elsewhere and one does not.

So we added a certification margin: the distance from the threshold, reported per entry. Ten of our 47 certified entries sit within 20% of the line. The tightest is H₁₀, certified at 9.977 mHa against a 10 mHa threshold — a margin of 0.2%.

The two fragile entries are flagged, not withdrawn. They reproduce exactly on the reference environment, which is what certification attests. What changed is that the fragility is now visible instead of implied.

A trap worth passing on

Midway through, we tried to predict which entries would fail by arithmetic: published error plus measured energy movement, compared to the threshold. It named four. We tested them. Two passed — their energy had moved toward the reference, so the error shrank: 9.283 → 8.405 mHa, and 7.917 → 5.684 mHa.

The movement is unsigned; it does not tell you the direction. Had we trusted the arithmetic we would have branded two healthy entries as fragile. Fragility gets established by running the check, never by inferring it.

What generalises

Little of this is specific to quantum chemistry.

• A broken checker looks like a broken command. Failures that surface as usage errors get attributed to the user, not to the tool. Ours hid in plain sight for months.
• Run the checker over everything, automatically. Spot-checking found nothing across many individual runs. One exhaustive pass found three bugs in an afternoon.
• Any benchmark with a hard threshold has entries just inside it whose status depends on the environment. If you publish a pass/fail label without a margin, you are hiding that.
• State the guarantee you can defend. It is tempting to let “reproducible” imply bit-identity. We measured what our own methods actually deliver and wrote that down instead.

All 54 entries, the sweep records, the measured envelopes and the tools are in the repository, including the runs that failed and the two entries that still do.