sigir-artifact-evaluation

SIGIR Artifact Evaluation

At SIGIR, "artifact" means something more specific than in most ML venues: the community's unit of exchange is the run file + qrels + index recipe, inherited from the TREC evaluation tradition. A SIGIR artifact is convincing when a stranger can rebuild your ranking, score it with standard tooling, and get your table. This skill covers packaging that artifact — and the routing decision that comes first.

Routing: artifact-in-paper vs Resources paper

SIGIR 2026 explicitly forbids double-dipping: the same dataset cannot be both a Resources track submission and the contribution of another paper. Decide ownership:

The anonymity asymmetry matters operationally: Resources reviewers may inspect the real, non-anonymized resource, while full-paper reviewers must see an anonymized mirror. Same artifact, two different packaging jobs.

The reviewer-runnable IR repository

Structure the repository around the evaluation chain, because that is how an IR reviewer will try to audit it:

repo/
  README.md            # 10-minute path: install -> retrieve -> evaluate -> Table 2
  environment.yml      # or Dockerfile; pin the retrieval toolkit version
  data/DOWNLOAD.md     # scripted fetch for public collections; never redistribute
  indexing/build.sh    # exact analyzer/tokenizer settings — silent nDCG movers
  runs/                # TREC-format run files behind every table row
  qrels/               # only if you created judgments; else pointers + checksums
  eval/score.sh        # ir_measures / trec_eval invocation with exact flags
  eval/significance.py # the paired test that produced the paper's p-values
  MANIFEST.md          # table-of-paper -> script -> run file mapping

Non-negotiables:

• Run files are the artifact. Ship the exact TREC-format runs behind every reported number; they let reviewers verify metrics without re-running GPUs.
• The index recipe is part of the method. Stemming, stopwords, max sequence length, and doc-splitting settings change scores; record them as code, not prose.
• Score with community tooling (trec_eval, ir_measures, ranx, or the toolkit's own eval) so numbers are checkable in one command.
• Checksums for derived data: qrels subsets, filtered corpora, sampled queries.

# The audit a reviewer (or you, pre-submission) should be able to run
conda env create -f environment.yml && conda activate repro
bash indexing/build.sh && bash eval/score.sh runs/ours.trec
python eval/significance.py runs/ours.trec runs/bm25.trec  # matches §5?

Judgments and collections as artifacts

If you built topics, judgments, or a corpus:

• Document the annotation protocol: assessor pool, guidelines, pay, agreement statistics (e.g., Cohen's or Krippendorff's), and adjudication.
• State pooling: which systems contributed to the judged pool and to what depth — unpooled dense-retrieval evaluation is a known validity trap reviewers probe.
• License explicitly (CC variants for data; note source-document terms separately) and include a datasheet: provenance, intended use, known biases, PII handling.
• For web/log-derived data, describe the privacy pipeline; "anonymized internally" without method is treated as unusable by careful reviewers.

Anonymized review packaging (full/short papers)

• Mirror the repo to an anonymous host; scrub commit history (fresh export, not a redacted clone), usernames in paths, and institution-specific cluster scripts.
• Keep the mirror small and runnable: reviewers grant minutes, not hours. The 10-minute README path decides whether the artifact helps or is ignored.
• Model checkpoints too large to host anonymously: ship the training script plus the exact seed/config, and say so plainly in the README.

Packaging failures reviewers actually hit

Observed failure modes, in descending frequency:

• The README's first command fails (missing environment.yml pin, absolute paths, CUDA assumptions) — the reviewer stops there and the artifact scores as absent.
• Run files present but not mapped to tables — without a MANIFEST, a reviewer cannot tell runs/final3.trec from runs/final3_fixed.trec.
• Redistributed data the license forbids (qrels, corpus slices) — a policy problem that outlasts the review.
• Evaluation script computes a nonstandard metric variant silently (e.g., a different gain function for nDCG) — the "numbers don't match" review comment.
• Anonymization done by deletion: the repo compiles but the interesting config was "removed for anonymity," which reads as hiding.

Post-acceptance hardening

• Replace the mirror with the public repository before camera-ready; mint an archival DOI (Zenodo or institutional) for the frozen state the paper describes.
• Register resources where the community looks: ir_datasets integration, a Hugging Face dataset card, or TREC-adjacent registries as fits the artifact.
• Badging: ACM defines artifact badges, but whether SIGIR applies them this cycle was not verifiable (待核实) — treat badges as optional polish, run-file hygiene as core.

Output format

[Artifact route] in-paper repo / Resources paper / Reproducibility track / release-only
[Runnable path] install->index->retrieve->score minutes: <n> (goal <=10 read + run start)
[Run-file coverage] tables backed by shipped runs: <k>/<n>
[Index recipe] scripted y/n; analyzer settings recorded y/n
[Judgment docs] protocol/agreement/pooling/license: complete / gaps <list>
[Anonymity mode] double-anonymous mirror / single-anonymous real repo
[Post-acceptance] DOI plan, ir_datasets/HF registration plan