ai-duel

AI Duel Simulation

Run AI-vs-AI game simulations to test decision quality, validate matchups, and catch regressions.

Quick Start

# Default: Red Aggro vs Green Midrange, 5 games, Medium difficulty
rtk cargo run --release --bin ai-duel -- client/public --batch 5

# Single verbose game (see every combat action and spell cast)
rtk cargo run --release --bin ai-duel -- client/public --seed 42 --difficulty VeryHard

# Batch with specific seed for reproducibility
rtk cargo run --release --bin ai-duel -- client/public --batch 20 --seed 1000 --difficulty Medium

# Full registered matchup suite in measurement mode
rtk cargo run --release --bin ai-duel -- client/public --suite --games 10 --seed 42 \
  --output target/duel-suite-results.json

# Compare two suite reports with paired-seed sign-test status
rtk cargo run --release --bin ai-duel -- compare crates/phase-ai/baselines/duel-suite.json \
  target/duel-suite-results.json

# Commander candidate-seat measurement
rtk cargo run --release --bin ai-duel -- client/public --commander-suite --games 8 --seed 42 \
  --difficulty Hard --baseline-difficulty Medium \
  --output target/commander-suite-results.json

CLI Options

Measurement And Gates

ai-duel uses AiConfig::into_measurement(seed) for single, suite, and Commander regression runs. Measurement mode disables wall-clock search budgets and bounds search by node/depth budgets, so outcomes are functions of the binary, config, matchup, and seed.

Use rtk cargo ai-gate for the normal regression gate. It runs the pinned suite against crates/phase-ai/baselines/, compares paired seeds, and reports FAIL/WARN/PASS with a binomial sign-test p-value. WARN means movement was observed but not significant enough to fail the gate.

When intentionally changing AI behavior, refresh the baseline in the same PR after review:

rtk ./scripts/refresh-ai-baseline.sh
rtk cargo ai-gate

Performance Guide

All times are release mode (--release). Debug mode is 5-10x slower.

Deck Configuration

The ai-duel binary resolves --matchup and --suite entries from crates/phase-ai/src/duel_suite/spec.rs. Inline starter/metagame deck builders live in crates/phase-ai/src/duel_suite/inline_decks.rs; snapshot decks live under crates/phase-ai/duel_decks/.

Available Matchups

Use --matchup NAME to select a preset. Use --list-matchups to see all options.

Starter decks (mono-colored, simple cards for baseline testing):

Metagame decks (real competitive lists from MTGGoldfish feeds, 100% engine coverage):

Changing Decks

To add new matchups, add a MatchupSpec in duel_suite/spec.rs. Use an inline builder only for small stable decks; prefer snapshot decks for real metagame coverage. Card names must match entries in client/public/card-data.json. Use jq 'keys[]' client/public/card-data.json | rg -i "card name" to find exact names.

To find high-coverage metagame decks for testing, check the feed data:

# List all feeds
ls client/public/feeds/

# Check a deck's card coverage against the engine
python3 -c "
import json
with open('client/public/card-data.json') as f:
    db = {k.lower(): v for k, v in json.load(f).items()}
with open('client/public/feeds/mtggoldfish-pioneer.json') as f:
    feed = json.load(f)
for deck in feed['decks']:
    sup = sum(1 for e in deck['main'] if e['name'].lower() in db)
    print(f'{sup}/{len(deck[\"main\"])} {deck[\"name\"]}')
"

Matchup Triangle (Expected Results)

The classic archetype triangle should hold:

• Aggro > Control — kills before control stabilizes
• Control > Midrange — removal + card draw outgrinds
• Midrange > Aggro — bigger creatures brick aggro attacks

Control decks improve more at higher difficulty levels (they need search to time removal correctly).

Mirror Match Testing

For testing AI quality independent of deck matchup advantage, use mirror matches (prowess-mirror, red-mirror, etc.). Win rates should be close to 50/50.

Commander Suite

--commander-suite loads four resolvable decks from the Commander feed, runs one candidate seat against three baseline seats, and rotates the candidate seat across P0-P3. Metrics are candidate win rate, survival turns, and elimination order. Use this as the first strength signal for multiplayer/Commander changes; it is slower than the two-player suite and should run nightly or on targeted AI changes rather than on every parser-only change.

Interpreting Results

Healthy signs:

• 0 draws/aborted games
• Games complete in 10-20 turns
• Win rates match expected archetype matchups
• Higher difficulty = longer games (smarter defensive play)

Warning signs:

• Any draws/aborted games → AI might be stuck in a loop
• Games > 30 turns → AI might not be attacking efficiently
• Same player always wins regardless of seed → deck balance issue
• Higher difficulty = worse results → search/evaluation regression

Verbose Output Patterns to Watch

When running single verbose games, look for:

• Self-targeting: "X deals N damage to X" — anti-self-harm policy failure
• Wasteful spells: Combat tricks cast outside combat, counterspells with empty stack
• Suicidal blocking: Blocking at low life when the block damage kills you
• Not attacking with lethal: Having lethal on board but not swinging
• Tapping out into lethal: Casting sorcery-speed when opponent has lethal on board

Community Scenario Fixtures

When a Discord #ai-suggestions thread includes a saved game-state zip, turn it into a deterministic AI scenario before landing the fix:

1. Download the zip and commit it under crates/phase-ai/fixtures/scenarios/.
2. Add an entry to crates/phase-ai/fixtures/scenarios/community-scenarios.json with the Discord thread id, archive name, and expected action assertion.
3. Run the fixture through crates/phase-ai/tests/community_scenarios.rs; it loads the same { "gameState": ... } export used by ai-bench-state, applies saved-state compatibility migrations, and evaluates the action in measurement mode with a fixed seed.

Related Files