siggraph-artifact-evaluation

SIGGRAPH Artifact Evaluation (Replicability Stamp)

SIGGRAPH does not run the ACM Artifact Review and Badging scheme that SIGSOFT and systems venues use. The computer-graphics community's equivalent is a community-run replicability stamp, earned after acceptance and independent of the review: the Graphics Replicability Stamp Initiative (GRSI, replicabilitystamp.org) and the related Code Replicability in Computer Graphics (CRCG, replicability.graphics). The distinction matters: a stamp certifies that a volunteer rebuilt your code and reproduced your paper's results, and archives that code for the community. Facts below trace to resources/official-source-map.md; confirm the current GRSI process before you package.

What the stamp is (and is not)

The GRSI has recognized graphics code since 2016 and is supported across the field's venues (ACM TOG, IEEE TVCG, Wiley CGF, Elsevier C&G, CAD/CAGD). CRCG has, since July 2020, focused specifically on checking whether SIGGRAPH papers' results are replicable. Neither is required for publication — but the stamp is the credible, checkable signal of a runnable contribution in this community.

What a volunteer actually does

Assume a graphics-literate volunteer clones your repository on their own machine and tries to regenerate a headline result from your paper. They are checking replicability of results, not just that the code compiles:

Design so the first headline result reproduces from a documented command on a clean checkout.

Packaging plan for a replicable graphics artifact

[Build]        a documented, pinned build (CMake/conda/Docker) that compiles on a clean machine;
               name exact compiler/CUDA/driver versions where GPU code is involved
[Assets]       ship (or give a stable download for) the scenes, meshes, textures, and weights the
               results need -- code without inputs cannot replicate a figure
[Determinism]  fix seeds; document tolerance for floating-point/GPU non-determinism; state which
               results are bit-exact vs perceptually-equal
[Mapping]      a table: paper figure/table -> command -> expected output image/metric/frame
[Timings]      report the hardware and the wall-clock the paper claims, so timings are checkable
[License]      an OSI-approved license so the code can be shared and reused
[Archive]      a Software Heritage snapshot (GRSI archives accepted code there) + a Zenodo DOI

Determinism is the graphics-specific hard part

Reproducing an image is not reproducing a number. Plan for it:

• State the comparison metric and tolerance. Volunteers cannot judge "looks right" — give PSNR/SSIM/LPIPS or a diff threshold against a bundled reference image.
• Pin the stochastic pieces. Monte Carlo renderers, stochastic simulations, and GPU reductions drift; seed them and document the residual non-determinism.
• Bundle reference outputs. Include the exact images/frames the paper reports so a volunteer compares against ground truth, not a re-render of their own.
• Separate "replicate the result" from "retrain the model." For learning-based work, ship weights and an inference path; full retraining is rarely the replication target.

Calibration

• The stamp is post-acceptance and optional; it is decided by the initiative's volunteers on their timeline, not by the SIGGRAPH committee, and not on the camera-ready deadline.
• GRSI and CRCG are related but distinct efforts; check each initiative's current submission instructions and supported-venue list before submitting your code.
• A stamp strengthens a paper's reach but is never a substitute for in-paper evidence — the review already happened without it.

Output format

[Target] Graphics Replicability Stamp (GRSI) / CRCG check / both
[Headline result] <figure/table the volunteer will reproduce>
[Clean-machine build] compiles + runs from documented command? yes/no
[Assets] scenes/meshes/weights shipped or stably linked? yes/no
[Determinism] seeds fixed + tolerance + reference outputs bundled? yes/no
[Claim mapping] <figure -> command -> expected output present?>
[Archive] Software Heritage snapshot + DOI + OSI license? yes/no
[Fixes before submitting code] <ordered>