August 30, 2026 ·

Reproduction as an artifact

August 30, 2026

A paper reports that a method achieves some result. The claim is the paper's contribution, but the claim is not the evidence. The evidence was a run: code executed on data, under a configuration, on a machine, at a moment. The paper is a compressed account of that run, written for a reader who will not perform it.

Compression is the point of a paper and also its failure mode. What survives into the text is what the authors judged necessary to convey the idea. What does not survive is everything they knew implicitly: the preprocessing that was obvious to them, the hyperparameter that mattered more than expected, the seed, the version of the library whose default changed the following spring. The reader inherits the claim without the conditions under which it held.

This gap is measurable and it has been measured. Monya Baker's 2016 survey of 1,576 researchers for Nature found that more than 70% had tried and failed to reproduce another group's experiment, and more than half had failed to reproduce one of their own. The Open Science Collaboration replicated 100 psychology studies and found that 97% of the original papers reported a statistically significant result while 36% of the replications did. Christian Collberg and Todd Proebsting went looking for the code behind 402 papers in computer systems and, within a thirty-minute budget, obtained and built it for 32.3% of them.

The last of those three is the one worth sitting with, because computer science is the field where reproduction should be cheapest. There is no reagent, no cohort, no instrument to recalibrate. The artifact is a program. And still, two papers in three did not yield a working build to a determined reader with the paper in hand.

What re-running a paper actually requires

The naive model of reproduction is: fetch the repository, run the script, compare the number. Anyone who has done it knows the naive model is wrong, and it is worth being precise about why, because the failure is structural rather than incidental.

Reproducing a computational result requires resolving a chain of underdetermined choices, each of which the paper may or may not fix:

Identity. Which artifact is the artifact? A paper often has a preprint, a camera-ready, and a repository, and the three disagree. The repository has a hundred commits after publication. Two people auditing the same paper must first agree on what they are auditing, or their disagreement about the result is really a disagreement about the object.

Availability. What did the authors actually release, and under what terms? Code, data, weights, preregistration, license. This is a factual question with a factual answer, including the answer "nothing", and it is separable from whether the result holds. A paper can be irreproducible because it is wrong, or because it is unavailable, and conflating the two is an error that helps no one.

The unwritten decisions. This is the substance. Every reproduction attempt makes dozens of choices the paper does not determine: how ties are broken, whether the metric is macro or micro averaged, which split the numbers in Table 2 refer to, what happens to the sample the filter drops. Each choice is defensible. Each choice moves the result. A reproduction that does not record these choices is not a reproduction anyone can check, because a reader cannot tell whether a discrepancy came from the paper or from the reproducer.

Execution. The environment, the versions, the hardware, the wall-clock and the budget. A result that only appears on eight particular accelerators is a different kind of claim from one that appears on a laptop, and the difference belongs in the record.

None of this is exotic. It is the ordinary work of the graduate student assigned to "just check this", and it takes weeks. The cost is why it is so rarely done, and the rarity is why the literature accumulates claims that nobody has independently exercised.

Why the verdict must decompose

The tempting output of an audit is a number. Reproducibility score: 0.62. It is tempting because it sorts, and a thing that sorts can rank labs, journals, and fields.

It is also close to useless, for a reason that has nothing to do with how the score is computed. A paper is not one claim. It is a set of claims of different kinds, resting on different evidence, failing in different ways. The main result reproduced but the ablation did not. The method works and the baseline was misconfigured, so the comparison is what fails. Figure 3 regenerates exactly; Figure 4 depends on data that was never released. Fuse these into a scalar and you get a number that is defensible in no particular direction, and that cannot be argued with because there is nothing specific in it to argue with.

So the unit of the verdict is the component, and each component gets one of a small set of outcomes. It reproduced. It did not. It could not be judged, for a stated reason: the data is not public, the licence forbids redistribution, the compute is out of reach, the claim is not the kind a re-run settles. That last category is not an embarrassment to be minimised. It is the part of the audit that tells a reader where the evidence stops, and an audit that reports it honestly is more useful than one that quietly folds it into a score.

The same argument applies to who says what. The agent's claim about what it reproduced, the transcript of what it actually did, and a human's sign-off are three different objects with three different warrants. Keeping them separate is what makes the result checkable. Fuse them and you have a number to trust; keep them apart and you have a result to check. Those are not the same product.

The transcript is the deliverable

If the reproduction is the evidence, then the record of the reproduction is what other people can actually use, and it should be treated as the primary output rather than as logging.

This inverts the usual emphasis. A conventional tool reports a conclusion and keeps the process in a debug log nobody reads. But the conclusion is the least reusable part: it is one bit, and a reader who disagrees with it can do nothing except disagree harder. The transcript is where the value is. It contains the decisions, in order, with what was known when each was made. A reader who thinks the reproduction chose wrong at step 40 can say so precisely, redo it from there, and produce a rival reproduction that differs in exactly one respect.

That is what makes an audit part of the scientific record rather than an opinion about it. A published reproduction should be a repository: the code that was written, the transcript of how it came to be written, the manifest of what was resolved, and the verdict, sitting together where the next person can fork them. Not a certificate. A starting point.

What this is for

The reproducibility literature has spent a decade establishing that the problem is real and expensive. The measurements are in. What has not changed much is the cost of doing something about any individual paper, which is still weeks of skilled attention, which is why the response has mostly been policy: checklists at submission, badges at acceptance, mandates from funders. Those are worth having. They also operate on the paper's description of its evidence rather than on the evidence.

The alternative is to make the re-run cheap enough to be routine, and to make its output a durable artifact rather than a verdict someone once reached. That is what ReplicHAI is: give it a paper, and it resolves the paper to one identity, records what the authors released, implements and executes the work, writes down every decision the paper left unwritten, and returns a verdict decomposed into components with the full transcript committed beside it. Reproductions its owner publishes go into a public registry, each in its own repository, so the audit is available to be checked, disputed, and built on by anyone who cares to.

We think that is the right shape for the thing. A reproduction is not a score and not an opinion. It is an artifact, and artifacts can be argued with.

References