kdd-experiments

KDD Experiments

Use this while the experimental plan is still cheap to change. A KDD empirical section answers four questions in order: does the method win, at what scale, at what computational price, and because of which mechanism? Papers that answer only the first question read as ML-flagship rejects retargeted at KDD — a pattern this venue's reviewers name openly.

The four-axis evidence plan

A missing axis should be a stated limitation, never a silent hole.

Data hygiene the practitioner-reviewers hunt for

• Temporal integrity: any data with a timestamp gets time-ordered splits. Random splits on temporal interaction data are the single most-caught flaw in KDD reviews — they leak future behavior into training.
• Leakage audit: feature construction must be causally valid at prediction time (no post-outcome aggregates, no target-derived normalizations).
• Popularity effects: for graph/recsys data, report performance stratified by node degree or item popularity at least once; aggregate wins that come entirely from the head of the distribution are a known illusion.
• Dataset provenance: name the snapshot/version; "the Twitter dataset" is not an identifiable object.

Baseline discipline

• Include the boring strong baselines. At KDD, a well-tuned gradient-boosting model, a popularity heuristic, or a classical index structure regularly embarrasses elaborate architectures — reviewers know it and check for their absence.
• Equalize tuning budgets and disclose them (kdd-reproducibility); a grid of 200 configs for yours vs defaults for theirs is a soundness objection, not a detail.
• Re-implementations must be validated: reproduce the original paper's reported number on its dataset before comparing on yours, and say so.

Ablation logging that isolates mechanisms

# ablations.py - one row per (variant, dataset, seed); the paper's ablation
# table is a groupby over this log, never hand-assembled.
VARIANTS = {
    "full":            dict(drift_weighting=True,  sketch_family=True),
    "-drift_weight":   dict(drift_weighting=False, sketch_family=True),
    "-sketch_family":  dict(drift_weighting=True,  sketch_family=False),
    "base":            dict(drift_weighting=False, sketch_family=False),
}
for name, flags in VARIANTS.items():
    for ds in DATASETS:               # each with row/edge counts in its metadata
        for seed in SEEDS:            # repeats where scale permits
            m = run(config(**flags), dataset=ds, seed=seed)
            log_row(variant=name, dataset=ds.name, seed=seed,
                    auprc=m.auprc, mem_mb=m.peak_mem, evps=m.throughput)

The point of the matrix: the paper's central claim ("the gain comes from the drift weighting") must be attributable from the log alone. Report efficiency columns in the ablation table too — a component that adds +0.4 quality for 3x memory is a different result from +0.4 for free.

ADS-track measurement design

Post-launch quantification is a desk-check item on the 2026 ADS CFP, so design the measurement, don't just harvest it:

• Prefer a controlled rollout (A/B or interleaving) with stated traffic share and duration; where only pre/post is possible, name the confounders in the window (seasonality, concurrent launches) and how they were handled.
• Define every online metric exactly once (numerator, denominator, window) and map each offline metric to the online metric it was supposed to predict — the offline-online correlation discussion is high-value ADS content.
• Report guardrail metrics (latency, cost, complaint rates), not only the success metric; practitioners on the committee ask what the win cost.
• If deployment was blocked, the CFP's exception path needs documented evidence of the blocker, not a hypothetical deployment story.

Vignette: auditing a recommendation paper's evidence

A draft claims a new sequential recommender beats five neural baselines on three datasets. The four-axis audit finds: quality covered; scale absent (largest dataset is 1M interactions — small for the claim "industrial-scale"); efficiency absent (training time never reported); mechanism partial (one ablation, but it removes two components at once). The hygiene audit finds random splits on timestamped data and no popularity stratification. Repair plan, ordered by review impact:

1. Rebuild splits time-ordered and rerun everything — a result that dies here was never real, and finding out pre-submission is the whole point.
2. Split the joint ablation into per-component toggles (the matrix pattern above).
3. Add one genuinely large public interaction dataset or delete the word "industrial-scale" from the paper.
4. Add a training-cost column to the main table; if the method is slower, say by how much and argue the trade.
5. Add the degree-stratified breakdown for the headline dataset.

Steps 3-5 are a week of compute; step 1 can invalidate the paper. Run it first.

Reporting floor

• Every stochastic table cell: repeat count and dispersion (IQR or std), or an explicit single-seed disclosure at the largest scales.
• Every dataset at first mention: cardinalities (users/items/edges/events), time span, and version or snapshot date.
• Every efficiency figure: hardware, software versions, and whether times include data loading.
• Every ablation row: identical budget and splits as the full method, or the row is not evidence.

Output format

[Axis coverage] quality/scale/efficiency/mechanism: <present-missing per axis>
[Split integrity] temporal-safe: yes/no; leakage audit: done/open items
[Baseline symmetry] tuning budgets equal + disclosed: yes/no
[Ablation matrix] mechanisms isolated: <list>; efficiency logged alongside: yes/no
[ADS measurement] design: A-B / pre-post / blocked-exception / N-A
[Decision-critical missing run] <the one experiment to do next>