eurosys-reproducibility

EuroSys Reproducibility

Use this while experiments are still running — reproducibility retrofitted in deadline week is transcription, not engineering. The venue context: EuroSys papers live on measured performance claims, the community runs a badge-granting artifact evaluation (sysartifacts.github.io), and SIGOPS has publicly digested five years of EuroSys AE lessons (sigops.org blog, 2025; rendered 2026-07-08). A paper whose numbers cannot be regenerated by its own authors three months later fails both review-time scrutiny and post-acceptance AE.

The provenance ledger

Keep one machine-readable record per reported number. Minimum fields:

Variance discipline

Single-run numbers are the most common silent reproducibility failure in systems evaluation:

• Repeat every headline measurement enough times to see its spread; report median plus an explicit dispersion measure (stdev, IQR, or min–max), and say in the caption which one it is.
• Isolate noise sources you control: pin frequencies, disable turbo where it distorts comparisons, note co-located load, randomize run order across systems so drift does not favor yours.
• Tail metrics (p99 and beyond) need far more samples than means; state the sample count whenever a tail latency is claimed.
• When a difference is within run-to-run spread, say so — EuroSys reviewers reward calibrated claims over uniform victory narratives.

Automation floor

The practical bar: any figure regenerates from raw data with one command.

# Layout that keeps figures honest
experiments/
  fig7_throughput/
    run.sh        # executes the sweep, writes results/*.csv with metadata header
    plot.py       # reads results/, emits fig7.pdf — no hand-edited numbers
    results/      # raw outputs, never overwritten, one dir per run timestamp
make fig7         # the only path by which fig7.pdf ever changes

If a plot was ever touched manually, its provenance is broken and the AEC will find the discrepancy before you do.

Availability statement, two audiences

• Review time (double-blind): describe what exists — "an anonymized repository containing the system, workload generator, and run scripts accompanies the submission" — without leaking the lab's identity through URLs, paths, or commit authors.
• Camera-ready / AE time: replace with the DOI-backed archive and the badge set being sought. Restricted traces (production data, partner NDAs) need an honest fallback: a synthetic generator calibrated to the trace's published statistics, with the calibration method described.

Restricted evidence, stated honestly

Some EuroSys evidence legitimately cannot ship — production traces under NDA, partner clusters, proprietary workloads. The honest pattern:

• Name the restriction and its scope precisely ("the ingestion trace from operator X cannot be released; its summary statistics are in Table 3").
• Ship a calibrated synthetic substitute and the calibration procedure, so external readers can approximate the regime.
• Keep at least one headline result on fully public inputs; a paper whose every number depends on unreleasable data asks reviewers for faith the venue does not trade in.
• Never let the availability paragraph imply more openness than the AEC will find; the badge process makes overstatement visible in print.

Pre-deadline reproducibility drill

One week before the paper gate, run the drill on a machine that never ran the experiments:

1. Clean checkout, environment build from the lockfile alone — record every undocumented step it turns out to need.
2. Regenerate two figures end to end: one cheap, one expensive.
3. Diff regenerated numbers against the draft's numbers; investigate any drift beyond the reported dispersion.
4. Fix the documentation, not just the outcome — the drill's product is the README the AEC will eventually read.

Quick self-test

1. Can a new student regenerate Figure 7 from a clean checkout in one command?
2. Does every table cell trace to a raw log file with hardware metadata?
3. Are repetition counts and dispersion visible for every performance claim?
4. Would the availability paragraph survive both anonymity and the AEC?

Output format

[Repro grade] regenerable / scripted-with-gaps / manual
[Ledger coverage] <numbers with full provenance / total reported numbers>
[Variance findings] <single-run claims, missing dispersion, undersampled tails>
[Workload provenance] <traces and generators with version + seed status>
[Availability draft] <review-time text and camera-ready plan>