OpenParable
Skip to content
← Back to skills

Percom Artifact Evaluation

Testing & Quality
percom-artifact-evaluation

Percom Artifact Evaluation is an Agent Skill for testing & quality published in the brycewang-stanford/Awesome-Journal-Skills repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use when packaging an IEEE PerCom sensing artifact and dataset for reproducibility and any badging (IEEE Open Research Objects / Results Reproduced, IEEE DataPort or Zenodo deposit), covering what a ubicomp evaluator checks first for human-subjects sensing data, cross-subject reproduction, de-identification, and honest degrees of reproducibility.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/PerCom-Skills/skills/percom-artifact-evaluation/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/percom-artifact-evaluation, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

PerCom Artifact Evaluation

Use this for reproducibility packaging. First, a cycle caveat: unlike SIGSOFT venues, PerCom has not historically run a mandatory formal artifact-evaluation track with a fixed badge set, and whether a given edition offers a reproducibility/badging track (e.g., IEEE Open Research Objects / Results Reproduced) is 待核实 — confirm on the current call. Regardless of whether a badge is offered, a well-packaged, de-identified sensing dataset and reproducible pipeline is a scored strength in the double-blind review and a lasting community contribution.

What "reproducible" means for a ubicomp sensing paper

Two deliverables, kept distinct:

  • The anonymized review package (at submission): dataset link and code scrubbed of owner, testbed, and lab identity, for the double-blind reviewers.
  • The public deposit (after acceptance): a de-identified dataset and code in a DOI-issuing archive under an open license — the version others cite and reuse, and the version any badge program evaluates.

What a ubicomp evaluator opens first

Claim typeFirst thing inspectedCommon failure caught
An activity/context recognizerThe script that regenerates cross-subject (LOSO) resultsOnly a pooled-accuracy script; no leave-one-subject-out path
A sensing datasetThe data itself + a datasheet (subjects, sensors, labels)Link present, data missing; no de-identification described
A deployed systemA demo on bundled sample dataOnly-runs-on-authors'-testbed; hardware not documented
A model resultTrained weights + inference on sample inputRequires the full raw dataset or private compute to run

Assume an evaluator gives your package a bounded time budget on a clean machine with no access to your sensors or subjects. Design for the first ten minutes — a demo on bundled, de-identified sample data — to succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
              "install these 40 things by hand"
[README]      one-screen orientation: what it is, install, run the demo, reproduce each claim,
              expected runtime and outputs
[Datasheet]   a dataset datasheet: subjects (count, relevant demographics), sensors (device,
              firmware, sampling rate, placement), labels + protocol, and known biases
[Mapping]     an explicit table: paper claim -> script -> expected result (with the LOSO split)
[Data]        the de-identified dataset itself (or documented restricted-access), not just a query
[Ethics]      IRB/consent status and the de-identification performed before release
[License]     an open, DOI-issuing deposit (IEEE DataPort, Zenodo) so others can reuse and cite

De-identification is the ubicomp-specific bar

Human-subjects sensing data leaks identity in ways code does not: raw audio/video, GPS traces, timestamps that pinpoint a home, and even accelerometer gait can re-identify. Before any release:

  • Remove or transform direct identifiers and re-identifying traces; document exactly what you did.
  • Confirm your consent and IRB approval permit public release of the de-identified data — some approvals do not, and then the honest move is restricted access with a documented request path.
  • Never ship a dataset that a subject did not consent to have published.

Worked vignette: a wearable-HAR dataset + recognizer

To make a HAR paper reproducible: ship a Docker image with the recognizer pre-built; a run_demo.sh that classifies on a small bundled, de-identified sample in under a minute; a reproduce/ directory whose scripts regenerate the leave-one-subject-out F1 table (not just a pooled number) from logged features; a datasheet listing subjects, sensor placement, and sampling rate; the de-identified dataset with a documented consent/IRB basis; and an open license with a DOI. State honestly which results are turnkey and which need the full (slow) training run.

Calibration

  • Whether a badge/reproducibility track runs, and which badges, is 待核实 per cycle — confirm on the current call rather than assuming a SIGSOFT-style scheme.
  • The DOI-issuing deposit and datasheet are portable value even when no badge is offered.
  • Anonymize the review package; de-identify (and confirm consent for) the public dataset — these are different obligations.

Output format

[Track status] formal reproducibility/badge track this cycle? yes/no/待核实
[Artifact role] anonymized review package / public de-identified deposit
[Contents] <recognizer/dataset/datasheet/scripts/ethics/license>
[Ten-minute test] does install + demo on bundled sample data succeed on a clean machine? yes/no
[Cross-subject reproduction] does a script regenerate the LOSO result? yes/no
[De-identification] documented + consent/IRB permits release? yes/no
[Fixes before deposit] <ordered list>