sosp-reproducibility

SOSP Reproducibility

Use this while experiments are being designed and run — months before the SOSP deadline, not during artifact evaluation. At SOSP the artifact process happens after acceptance (see sosp-artifact-evaluation), which means reproducibility discipline has a different job pre-submission: it protects you. Reviewers probe numbers during a three-month review cycle and a response phase in which new experiments are forbidden; the only defensible paper is one whose every number can be traced to a logged, re-runnable measurement of a pinned system.

The OS-research twist: your system changes the platform

An operating-systems artifact often is the environment — a modified kernel, a new scheduler, an interposed I/O path. That collapses the usual app/platform separation and creates specific hazards:

• A rebased kernel patch series can silently change baseline behavior; record the exact base commit and the full patch stack for every run, including baseline runs.
• Firmware, microcode, and mitigations (for example, speculative-execution mitigations) can dominate syscall-heavy microbenchmarks; log them and hold them constant across system and baseline.
• Frequency scaling, turbo states, and NUMA placement move tail latencies by tens of percent; fix the policy, and record it rather than assuming defaults.

Capture the environment mechanically

Hand-written "Experimental Setup" sections drift from reality. Generate the facts:

#!/usr/bin/env bash
# capture-env.sh — run on every experiment node, archive with the run's results
{
  uname -a; cat /etc/os-release | head -2
  cat /proc/cmdline                          # mitigations, isolcpus, hugepages
  lscpu | grep -E 'Model name|Socket|NUMA|MHz'
  free -h | head -2; lsblk -d -o NAME,MODEL,ROTA,SIZE
  ip -br link; ethtool eth0 2>/dev/null | grep -E 'Speed|Duplex'
  git -C "$REPO" rev-parse HEAD; git -C "$REPO" status --porcelain
} > "env-$(hostname)-$(date +%Y%m%dT%H%M%S).txt"

Archive one such file per node per experiment batch, next to the raw results. When a reviewer asks eleven weeks later whether the baseline ran with the same mitigations, the answer is a file, not a memory.

One command per figure

The standard that survives review pressure: every figure and table in the paper is produced by a script that reads only archived raw logs. No spreadsheet steps, no hand-transcribed numbers. This is also what makes the response phase survivable — you can re-check any reviewer-doubted number against raw data without re-running anything, which is the only kind of "checking" the response rules allow.

Variance is a first-class result

Systems effects live in distributions. Reproducibility at SOSP includes making the noise floor explicit:

• Report repetitions and the spread (percentiles or CIs), especially for tail-latency claims — a p99 from one run of one trial is folklore, not a measurement.
• Distinguish sources: run-to-run jitter, node-to-node hardware variation, and time-of-day effects on shared testbeds. If experiments ran on a shared cluster, say so and quantify what that cost in variance.
• Decide the warm-up policy (discard first N iterations? cold-start included?) once, document it, and apply it uniformly; asymmetric warm-up between system and baseline is a classic self-inflicted review wound.

Traces and workloads you cannot publish

Production traces make SOSP evaluations compelling and reproductions hard. The honest pattern: characterize the private trace in the paper (size, arrival statistics, skew, whatever drives the result), release a synthetic generator matched to those statistics, and show at least one headline experiment where synthetic and real traces agree in trend. Flag in the paper which results are re-runnable and which are documented-only — the same tiering the artifact evaluation will ask for later.

Output format

[Repro posture] traceable / partially / folklore
[Environment] capture automated? mitigations+kernel pinned for baseline too?
[Figure pipeline] all one-command? exceptions: <list>
[Variance] repetitions, spread reported, warm-up policy uniform?
[Private data] trace characterization + generator plan
[Gap list] <ordered fixes before the freeze>