# Sigmetrics Reproducibility

> Sigmetrics Reproducibility is an Agent Skill for testing & quality published in the brycewang-stanford/Awesome-Journal-Skills repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use when strengthening ACM SIGMETRICS reproducibility, covering proofs and their assumptions as reproducible artifacts, seeded simulators whose figures regenerate and match the analysis, measurement/trace provenance, claim-to-evidence mapping, honest degrees of reproducibility, and consistency between what the paper proves/measures and what the artifact contains.

- Skill name: `sigmetrics-reproducibility`
- Category: [Testing & Quality](https://www.openparable.com/skill/testing-quality/1)
- Source: https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/SIGMETRICS-Skills/skills/sigmetrics-reproducibility/SKILL.md
- Repository: brycewang-stanford/Awesome-Journal-Skills (805 GitHub stars when collected)

## SKILL.md

# SIGMETRICS Reproducibility

Use this before submission and again before the POMACS camera-ready. SIGMETRICS reproducibility has
a distinctive shape: the "artifact" is often a **proof plus a simulator plus a trace**, not only
running code. The goal is that a competent reader could re-derive your bound, re-run your simulation
to the same curves, and re-analyze your measurement to the same conclusions.

## Evidence map

- Map each theorem, bound, and reported number to a **verifiable location** — a proof in an
  appendix, a figure regenerated from a seeded simulation, or a script that turns the trace into the
  table.
- For analytic results, give the **full derivation and every assumption**; a reader should be able
  to check the proof and see which assumptions each step uses.
- For simulations, ship a **seeded simulator** whose scripts regenerate each figure *and* overlay
  the analytic prediction, so a reviewer sees model and measurement agree.
- For measurement studies, document the trace source, collection window, sanitization, and the
  processing scripts; archive the processed dataset or document access.
- Keep the paper and the artifact **consistent**: a p99 number in the PDF that no simulator run
  reproduces is the contradiction reviewers read as carelessness.

## Reproducibility-claim audit

| Claim in the paper | Weak reproducibility answer | SIGMETRICS-ready answer |
|---|---|---|
| "Theorem 1 bounds the tail" | Proof sketch only | Full proof (appendix) + a simulation that matches the analytic curve |
| "We simulate policy X" | "Simulator available on request" | Seeded simulator + scripts that regenerate each figure from logged runs |
| "We measured system Y" | "Data on request" | Processed dataset (or documented access) + provenance + processing scripts |
| "The learner has low regret" | Empirical curve only | Regret proof + code plotting empirical regret against the bound |

"Available on request" is treated as *not available*; convert every such line into a concrete,
anonymized artifact or an explicit, justified exception (e.g. a proprietary trace, with the
methodology fully documented).

## Provenance and determinism pinning

```text
[Proof]       state every assumption; give the full derivation; note which lemmas each step needs
[Simulation]  log seeds; state steady-state/warm-up handling; make figures regenerate deterministically
[Measurement] pin the trace source, collection window, sanitization; archive processed data
[Compute]     state hardware, runtime, and number of independent runs so a reader can size a rerun
[Agreement]   ship the overlay of analysis vs. simulation so the match is reproducible, not asserted
```

## Degrees of reproducibility (state the one you achieved)

- **Turnkey:** one documented command regenerates each figure/table from logged simulation runs and
  reproduces the analytic overlay.
- **Scripted:** scripts exist but require documented manual steps or access to a restricted trace.
- **Descriptive:** proofs and methodology detailed enough that a competent reader could rebuild the
  pipeline.

For SIGMETRICS, aim turnkey for anything a reviewer might rerun quickly (a simulation regenerating a
figure, a script producing a table); a proprietary industrial trace may stay scripted with access
documented, but the *methodology and the analysis code* should still be turnkey.

## Vignette: a queueing-theory-plus-measurement paper

Consider a paper with a scheduling theorem and a trace-driven evaluation. Its reproducibility spine:
the full proof with stated assumptions in an appendix; a seeded simulator whose notebook regenerates
the analysis-vs-simulation figure; the trace-processing scripts with pinned provenance; the
anonymized processed dataset (or documented access to a restricted one); and the analysis notebooks
that turn logged runs into the paper's tables — plus one honest sentence about any assumption that
only approximately holds and how §6 bounds it.

## Consistency and camera-ready pass

- Before submission: every reported number traces to a proof, a logged simulation run, or a
  measurement script; the artifact is anonymized (no owner strings, cluster paths, group names).
- Before camera-ready: swap anonymized links for a permanent, DOI-issuing archive, and align the
  artifact with any ACM badges you are pursuing (`sigmetrics-artifact-evaluation`).

## Output format

```text
[Claim inventory] <claim -> proof / simulation run / measurement script>
[Reproducibility] concrete / vague / missing, per claim
[Provenance gaps] <proof assumptions stated? seeds logged? trace provenance pinned?>
[Reproducibility level] turnkey / scripted / descriptive, stated honestly
[Paper fixes] <must appear in the PDF/appendix>
[Artifact fixes] <additions before upload>
```

---

Source: https://www.openparable.com/skill/testing-quality/sigmetrics-reproducibility (OpenParable). Cite the canonical URL above.
