Siggraph Artifact Evaluation
Testing & Qualitysiggraph-artifact-evaluationSiggraph Artifact Evaluation is an Agent Skill for testing & quality published in the brycewang-stanford/Awesome-Journal-Skills repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Use when pursuing the Graphics Replicability Stamp (GRSI) or Code Replicability in Computer Graphics (CRCG) recognition for an accepted SIGGRAPH / TOG paper, covering how graphics replicability differs from ACM artifact badging, what volunteers actually run, deterministic result reproduction, Software Heritage archiving, and the separate post-acceptance timing.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/SIGGRAPH-Skills/skills/siggraph-artifact-evaluation/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/siggraph-artifact-evaluation, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
SIGGRAPH Artifact Evaluation (Replicability Stamp)
SIGGRAPH does not run the ACM Artifact Review and Badging scheme that SIGSOFT and systems
venues use. The computer-graphics community's equivalent is a community-run replicability
stamp, earned after acceptance and independent of the review: the Graphics Replicability
Stamp Initiative (GRSI, replicabilitystamp.org) and the related Code Replicability in Computer
Graphics (CRCG, replicability.graphics). The distinction matters: a stamp certifies that a
volunteer rebuilt your code and reproduced your paper's results, and archives that code for the
community. Facts below trace to resources/official-source-map.md; confirm the current GRSI
process before you package.
What the stamp is (and is not)
| Graphics Replicability Stamp (GRSI) | ACM Artifact Badges (other venues) | |
|---|---|---|
| Who runs it | Volunteer researchers from the graphics community | The venue's artifact evaluation committee |
| What it certifies | Your code replicates the paper's results | Available / Functional / Reusable / Reproduced |
| When | Post-acceptance, on the initiative's own schedule | Post-acceptance, on the venue's AE deadline |
| Archiving | Software Heritage snapshot + long-term ID (since 2023) | DOI-issuing archive |
| Scope | Independent of SIGGRAPH's review; optional recognition | Tied to the venue's AE track |
The GRSI has recognized graphics code since 2016 and is supported across the field's venues (ACM TOG, IEEE TVCG, Wiley CGF, Elsevier C&G, CAD/CAGD). CRCG has, since July 2020, focused specifically on checking whether SIGGRAPH papers' results are replicable. Neither is required for publication — but the stamp is the credible, checkable signal of a runnable contribution in this community.
What a volunteer actually does
Assume a graphics-literate volunteer clones your repository on their own machine and tries to regenerate a headline result from your paper. They are checking replicability of results, not just that the code compiles:
| Contribution type | What the volunteer reproduces | Common failure caught |
|---|---|---|
| A rendering technique | A converged image matching a paper figure, within tolerance | Non-deterministic output; missing scene assets |
| A geometry/mesh method | A processed mesh matching the reported statistics | Hard-coded absolute paths; unshipped input meshes |
| A simulation | A representative frame/sequence from the paper | Unseeded RNG; platform-specific solver drift |
| A learning-based method | A result from released weights, not retraining from scratch | Weights absent; inference needs undocumented data |
Design so the first headline result reproduces from a documented command on a clean checkout.
Packaging plan for a replicable graphics artifact
[Build] a documented, pinned build (CMake/conda/Docker) that compiles on a clean machine;
name exact compiler/CUDA/driver versions where GPU code is involved
[Assets] ship (or give a stable download for) the scenes, meshes, textures, and weights the
results need -- code without inputs cannot replicate a figure
[Determinism] fix seeds; document tolerance for floating-point/GPU non-determinism; state which
results are bit-exact vs perceptually-equal
[Mapping] a table: paper figure/table -> command -> expected output image/metric/frame
[Timings] report the hardware and the wall-clock the paper claims, so timings are checkable
[License] an OSI-approved license so the code can be shared and reused
[Archive] a Software Heritage snapshot (GRSI archives accepted code there) + a Zenodo DOI
Determinism is the graphics-specific hard part
Reproducing an image is not reproducing a number. Plan for it:
- State the comparison metric and tolerance. Volunteers cannot judge "looks right" — give PSNR/SSIM/LPIPS or a diff threshold against a bundled reference image.
- Pin the stochastic pieces. Monte Carlo renderers, stochastic simulations, and GPU reductions drift; seed them and document the residual non-determinism.
- Bundle reference outputs. Include the exact images/frames the paper reports so a volunteer compares against ground truth, not a re-render of their own.
- Separate "replicate the result" from "retrain the model." For learning-based work, ship weights and an inference path; full retraining is rarely the replication target.
Calibration
- The stamp is post-acceptance and optional; it is decided by the initiative's volunteers on their timeline, not by the SIGGRAPH committee, and not on the camera-ready deadline.
- GRSI and CRCG are related but distinct efforts; check each initiative's current submission instructions and supported-venue list before submitting your code.
- A stamp strengthens a paper's reach but is never a substitute for in-paper evidence — the review already happened without it.
Output format
[Target] Graphics Replicability Stamp (GRSI) / CRCG check / both
[Headline result] <figure/table the volunteer will reproduce>
[Clean-machine build] compiles + runs from documented command? yes/no
[Assets] scenes/meshes/weights shipped or stably linked? yes/no
[Determinism] seeds fixed + tolerance + reference outputs bundled? yes/no
[Claim mapping] <figure -> command -> expected output present?>
[Archive] Software Heritage snapshot + DOI + OSI license? yes/no
[Fixes before submitting code] <ordered>