OpenParable
Skip to content
← Back to skills

Icml Artifact Evaluation

Productivity
icml-artifact-evaluation

Icml Artifact Evaluation is an Agent Skill for productivity published in the brycewang-stanford/Awesome-Journal-Skills repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use when packaging ICML artifacts - code, data, model weights, simulators, benchmarks, proof scripts, notebooks, anonymous repositories, and supplementary code/data ZIPs - for both the double-blind review package and the public release that accompanies accepted PMLR papers. Use when checking anonymity, decision relevance, licensing, and the OpenReview code URL field under current ICML rules.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/ICML-Skills/skills/icml-artifact-evaluation/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/icml-artifact-evaluation, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

ICML Artifact Evaluation

ICML does not let artifacts sit outside the paper's scientific argument. Reproducibility and code availability are explicitly considered in decision-making, and accepted submissions may publish the original supplementary material on OpenReview.

Review-stage package

  • Decide whether the artifact is code, data, model weights, simulator, benchmark, proof script, notebook, or supplementary manuscript.
  • Anonymize authors, repository ownership, filenames, commit history, logs, model cards, dataset cards, licenses, and personal paths.
  • If using an anonymous repository, put it on a branch that will not change after the submission deadline.
  • Put critical evaluation material in the paper body, not only in supplement. Reviewers decide whether to consult appendices or supplementary material.
  • Provide minimal commands, environment details, expected runtime, hardware assumptions, and result mapping.

Public-release package

  • Because accepted original supplementary material may become public, review-stage artifacts should not contain private, illegal, or unreleasable content.
  • For camera-ready, final supplementary material is not uploaded separately; code/data should move to a public repository or archive and be linked in the paper/OpenReview code URL field.
  • Add clear licenses and persistent identifiers when possible.

Anonymity leak checklist

ICML double-blind review means a single deanonymizing artifact can trigger a desk reject, so audit the package the way an adversarial reviewer would.

Leak vectorWhere it hidesMitigation
Repo ownershipAnonymous-repo account name, commit authorUse a fresh anonymized host, strip git history
File metadataPDF author field, notebook kernel, model cardClear metadata, rename author paths
Hard-coded pathsCluster usernames in scripts and logsReplace with placeholders before zipping
External linksPersonal site, non-anonymous URL, shortenerRemove or route through an anonymous mirror

Worked vignette: optimizer artifact package

A paper shipping an adaptive optimizer includes training scripts, a pretrained checkpoint, and a proof-checking notebook. The review package gives minimal commands, expected runtime, and the hardware assumption so a reviewer can map a command to a benchmark number, while the checkpoint filename and the notebook kernel are scrubbed of the lab name. Because accepted ICML supplements can become public, the team confirms the checkpoint is releasable and the license is stated before the deadline, then plans the public repository and OpenReview code URL for camera-ready.

Output format

[Artifact role] code / data / model / benchmark / proof / none
[Review package] sufficient / incomplete / unsafe
[Anonymity risks] <metadata, repo, filenames, links>
[Decision relevance] <why reviewers need it>
[Public release plan] <repo/archive/license/code URL>