OpenParable
Skip to content
← Back to skills

Aistats Experiments

Testing & Quality
aistats-experiments

Aistats Experiments is an Agent Skill for testing & quality published in the brycewang-stanford/Awesome-Journal-Skills repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use when designing or auditing AISTATS experiments, simulations, baselines, statistical tests, uncertainty estimates, ablations, random seeds, hyperparameters, compute, dataset handling, and claim-to-evidence fit, with emphasis on experiments that validate theorems rather than chase leaderboards.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/AISTATS-Skills/skills/aistats-experiments/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/aistats-experiments, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

AISTATS Experiments

Use this before submission when the empirical or simulation story is not yet locked.

Experiment audit

  • Map each empirical claim to a table, figure, simulation, ablation, or robustness check.
  • Include baselines that represent both ML practice and relevant statistical methods.
  • Separate synthetic simulations that validate assumptions from real-data experiments that show practical relevance.
  • Report uncertainty for stochastic results: repeated runs, standard errors, confidence intervals, paired tests, or bootstrap intervals when appropriate.
  • Report dataset splits, preprocessing, metrics, hyperparameter search ranges, final chosen settings, selection criteria, random seeds, hardware, software versions, and runtime.
  • Add ablations for the mechanism, not just cosmetic variants.
  • Audit for leakage, selection bias, multiple-comparison issues, and mismatch between theoretical assumptions and empirical setup.

What experiments are for at this venue

  • AISTATS experiments exist to validate theory, not to win leaderboards. One focused simulation confirming a predicted rate outweighs five extra benchmark datasets.
  • The strongest design triad: a synthetic study where assumptions hold exactly, a study where they are deliberately violated, and a real-data study showing practical behavior.
  • Reviewers, frequently statisticians, check whether the empirical regime — sample size, dimension, noise level — matches the asymptotic regime of the theorems. A bound proven as n grows but tested only at n = 500 invites the question of relevance.

Theory-validation design table

Theoretical claimMatching experimentReject pattern avoided
Convergence rate in nLog-log error versus n with fitted slope"Rates asserted but never plotted"
Confidence-interval coverageEmpirical coverage across many replications"Nominal 95 percent never verified"
Regret boundCumulative regret versus horizon, with the bound curve overlaid"Bound and trajectory never compared"
Robustness to misspecificationViolation-severity sweep"Guarantees hold under assumptions the experiments quietly break"

Vignette: a kernel conditional independence test

Suppose the paper proves finite-sample type-I error control under a boundedness assumption. The matching plan: simulate under the null at several sample sizes to verify size, sweep dependence strength for power curves, then inject heavy-tailed noise that breaks boundedness to map degradation — every panel tied to a numbered theorem or remark.

Statistical reporting floor

  • Replication counts and seeds for every stochastic figure; captions must say whether bars are standard errors, confidence intervals, or quantiles.
  • Report the compute actually consumed rather than vague feasibility language.

Output format

[Experiment readiness] strong / adequate / weak
[Claim -> evidence map] <claim: table/figure/simulation>
[Missing statistical evidence] <uncertainty/test/seed/baseline>
[Reproducibility gaps] <hyperparameters/compute/data/code>
[Decision-critical next run] <one experiment or simulation>