acl-reproducibility

ACL Reproducibility

Use this before an ARR deadline and again at camera-ready. At ACL the reproducibility instrument is the Responsible NLP checklist: it is mandatory, reviewers read it alongside the paper, and ARR policy makes incorrect or misleading checklist content a desk-rejection ground. Treat it as a claims audit, not paperwork.

The checklist as a claims audit

• Section A: a real Limitations discussion and a risks discussion — reviewers are told honest limitations must not be penalized, so under-disclosing is strictly worse than disclosing.
• Section B: every dataset and model you used needs citation, version, license, and intended-use consistency (see acl-artifact-evaluation).
• Section C: computational experiments — parameters, budget, infrastructure, hyperparameter search, and descriptive statistics with error bars.
• Section D: human annotators/participants — instructions, pay, consent, ethics-board status, demographics where relevant.
• Section E: AI assistants used in research, coding, or writing.

Every "yes" answer should carry a section/appendix pointer; every "N/A" should survive a hostile reading of the paper.

Reporting floor for the modern NLP paper

Contamination and leakage auditing

• State which evaluation sets could plausibly appear in pretraining corpora and what you did about it: n-gram overlap scans, canary checks, dataset release date vs model cutoff reasoning.
• For benchmarks you release, record a creation date and content hash so future contamination is auditable.
• For claimed generalization, verify the test languages/domains genuinely weren't leaked through translation or paraphrase of training data.

Variance discipline

• Single-run leaderboard deltas are the classic ACL review complaint. Report mean and deviation over multiple seeds or prompt paraphrases, and say in the caption what the interval is.
• When compute makes many runs impossible, say so, quantify what you could (e.g., variance on the smallest model), and scope claims accordingly — checklist Section C expects the compute budget stated either way.

Consistency sweep before submission

1. Grep the paper for every number that a checklist item claims exists (error bars, splits, licenses, pay). Missing → fix paper or answer.
2. Check the supplement actually contains what Sections B/C reference.
3. Confirm Limitations mentions the weaknesses your own experiments exposed; reviewers notice when the Limitations section dodges the obvious one.
4. Re-answer Section E honestly after the final writing pass — late-stage AI-assisted rewriting counts.

Degrees of reproducibility to declare

turnkey    : one script re-scores released outputs / reruns the pipeline
scripted   : code + configs released; needs GPUs, keys, or gated data
descriptive: enough prose + prompts that a motivated lab could rebuild it
closed     : hinges on private data or deprecated APIs — say so in Limitations

Declare the level you actually achieve. At ACL, releasing model outputs is the cheap trick that upgrades many LLM papers from descriptive to turnkey, because re-scoring needs no compute.

Prompt-disclosure block that satisfies reviewers

A reusable appendix pattern for each prompted experiment:

Experiment: Table 3, zero-shot NLI
Model: <name + exact version/snapshot + access date>
Decoding: temperature=0.0, top_p=1.0, max_tokens=16
Prompt (verbatim, incl. whitespace):
  "Premise: {premise}\nHypothesis: {hypothesis}\n
   Answer entailment, neutral, or contradiction:"
Paraphrases: 5 variants (App. D.2); reported number = mean over variants
Post-processing: first-token match, case-insensitive; ties -> neutral
Failures: non-parseable outputs counted as errors (2.3% of calls)

The last two lines — parsing rules and non-parseable handling — are where most "we could not reproduce the number" disputes actually originate.

Cheap wins ranked by effort

1. Release model outputs alongside code (near-zero cost, enables re-scoring).
2. Log and report seeds + run counts in every caption while runs are fresh.
3. Pin dataset versions/commits in the bibliography and README now, not at camera-ready when the version has silently moved.
4. Write the compute paragraph (GPUs, hours, total runs incl. failed) the week the experiments finish.
5. Save the exact evaluation-script commit used for headline numbers.

Output format

[Checklist status] consistent / gaps found / contradicts paper
[Section-by-section] <A/B/C/D/E: pass or missing items>
[LLM disclosure] <prompts/decoding/version/date status>
[Contamination stance] <audit done / reasoned / unaddressed>
[Variance reporting] <runs, intervals, caption clarity>
[Fixes] <paper edits vs supplement additions, ordered>