dac-experiments

DAC Experiments

Use this before the November deadline when the evaluation is not yet locked. At DAC the evaluation is the paper: reviewers are EDA practitioners who decide acceptance mostly on whether the QoR comparison is fair, standard, and reproducible. The organizing principle is measured design quality against the strongest baseline on recognized benchmarks — not novelty in the abstract.

Evaluation audit

• Use standard benchmark suites. Match the suite to the task: ISPD placement/routing contests, the EPFL combinational benchmark suite for logic synthesis, ISCAS'85/'89 and ITC'99 for test/verification, the TAU contests for timing, CircuitNet/OpenABC-D and similar for ML-for-EDA, and OpenROAD / OpenROAD-flow-scripts for full-flow experiments. A private-benchmark-only evaluation is a scored weakness.
• Compare against the true state of the art, tuned with a documented, equal effort. An untuned or outdated baseline is the most common DAC reject cause; the assigned reviewer often is the author of the stronger tool you skipped.
• Report the whole suite, not a subset. Per-benchmark tables with the full circuit set; a cherry-picked average invites "what happened on the circuits you dropped?"
• Report runtime and scalability, not just quality. EDA reviewers care whether the method scales to realistic design sizes (millions of cells), so include the largest benchmarks and the compute used.
• Ablate the mechanism. Isolate where the gain comes from — remove your key component and show the QoR degrade — so the reviewer can attribute the improvement to your idea, not to tuning.
• Design threats in. Know before you run which designs, PDKs, or corners limit generality, and instrument to bound them.

Claim-to-evidence design table

PPA and QoR reporting floor

• Report the full PPA picture: a wirelength or timing win that silently costs area or power is not a win — show the trade-off.
• Give per-benchmark numbers and the aggregate; state the geometric-mean convention you use.
• Report runtime for both your method and the baseline on the same hardware, and the hardware.
• For stochastic methods (simulated annealing, RL-based flows) report variance across seeds/runs, not a single lucky run.

Contamination-aware ML-for-EDA evaluation

When a learner is in the loop, the reviewer's first questions are about leakage and fairness:

[Split integrity]  train and test on DIFFERENT designs/netlists; never leak a test design into
                   training. Report the split explicitly.
[Baseline]         compare against the strong non-ML tool (analytic placer, classical STA) AND the
                   prior-ML method, not just an untrained control
[Generalization]   evaluate on designs/technology nodes unseen in training; ML-for-EDA that only
                   works on its training distribution is a scored weakness
[Determinism]      report seeds and variance; a single run is not evidence for an RL flow
[Cost honesty]     report training cost and inference cost; a method needing per-design retraining
                   must say so
[Data provenance]  name the dataset (CircuitNet, OpenABC-D) and version; cache generated data

Vignette: evaluating a new global router

A paper claims a router that cuts congestion at equal wirelength. The matching plan: run on the full ISPD routing benchmark set (not a subset); compare against the strongest published router tuned to equal effort; report per-benchmark wirelength, DRC/overflow, and runtime on stated hardware; include the largest circuits to show scaling; ablate the congestion-aware component to show it, not parameter tuning, drives the gain; and state external validity (technology node, macro density) as a bounded threat — every number traceable to a logged, re-runnable flow.

Output format

[Evaluation readiness]  strong / adequate / weak
[Benchmarks]            standard suite(s) named + full set reported? yes/no
[Baseline fairness]     strongest SOTA, tuned, equal effort, on same hardware? yes/no
[QoR completeness]      full PPA + runtime + variance reported? yes/no
[Ablation]              mechanism isolated from tuning? yes/no
[ML leakage]            train/test designs disjoint + unseen-node generalization? yes/no/NA
[Decision-critical run] the one experiment that would most strengthen the case