bat-adhoc

BAT - Bot Acceptance Testing

Bot acceptance testing validates that MCP tools work correctly from a real AI agent's perspective. You design test scenarios dynamically, run them via tests/uat/run_uat.py, and evaluate results.

When to Use BAT

• PR validation: Test that tool changes work correctly from an agent's perspective
• Regression detection: Compare behavior between branches
• Integration verification: Ensure MCP tools work end-to-end with real agent CLIs

Workflow

1. Analyze the change: Read the diff, identify which tools are affected
2. Design scenario: Generate a scenario JSON with setup/test/teardown prompts
3. Run the script: Pipe the scenario to python tests/uat/run_uat.py
4. Evaluate summary: Check all_passed per agent. If true, you're done.
5. Dig deeper on failure: Read results_file for full output, stderr, raw JSON
6. Regression check: If test fails, re-run with --branch master to compare

Output Structure

The runner returns a concise summary to stdout (saves context when all passes):

{
  "results_file": "/tmp/bat_results_abc123.json",
  "agents": {
    "gemini": {
      "all_passed": true,
      "test": {
        "completed": true,
        "duration_ms": 8100,
        "exit_code": 0,
        "num_turns": 5,
        "tool_stats": { "totalCalls": 4, "totalSuccess": 4, "totalFail": 0 }
      },
      "aggregate": {
        "total_duration_ms": 15300,
        "total_turns": 12,
        "total_tool_calls": 9,
        "total_tool_success": 9,
        "total_tool_fail": 0
      }
    }
  }
}

• Phase stats: num_turns, tool_stats (per phase) for fine-grained comparison
• Aggregate stats: Total counts across all phases for overall efficiency comparison
• On failure: also includes output and stderr for diagnosis
• Full results: raw JSON, complete output always available at results_file

Scenario Design Guidelines

• setup_prompt: Create any entities/state the test needs
• test_prompt: Exercise the tools being tested, ask the agent to report results clearly
• teardown_prompt: Clean up created entities
• Keep prompts focused - each scenario tests ONE behavior
• Ask the agent to report: what succeeded, what failed, any unexpected behavior

Example: Testing Error Signaling

cat <<'EOF' | python tests/uat/run_uat.py --agents gemini
{
  "setup_prompt": "Create a test automation called 'bat_error_test' with a time trigger at 23:59 and action to turn on light.bed_light.",
  "test_prompt": "Try to get automation 'automation.nonexistent_xyz'. Report if the tool signaled an error or returned a normal response. Then get automation 'automation.bat_error_test' and report its structure.",
  "teardown_prompt": "Delete automation 'bat_error_test' if it exists."
}
EOF

Regression Comparison Workflow

Full BAT comparison (recommended):

1. Pull latest master: git fetch origin master && git checkout master && git pull
2. Run on master: Save scenario to file, run and save results
3. Switch to branch: git checkout feat/my-branch
4. Run on branch: Run same scenario, compare stats

Compare these metrics:

Primary (decide pass/fail on these):

• Task completion: Did both pass? Any new failures?
• Accuracy: Check agent output quality - did it understand the task correctly?
• Tool success rate: Compare aggregate.total_tool_calls vs total_tool_fail

Secondary (report but don't decide on these alone):

• Tool call count: Compare aggregate.total_tool_calls, aggregate.total_turns — directional signal, not conclusive (agent exploration varies between runs)
• Duration: Compare aggregate.total_duration_ms — noisy due to network, cache misses, server load. Only flag large (>2x) regressions.

Robustness tip: Ask the same task in different ways (variation testing) to check if results are consistent across phrasings.

Quick comparison (single command):

# Test the PR branch
echo '{"test_prompt":"..."}' | python tests/uat/run_uat.py --branch feat/tool-errors --agents gemini

# Compare against master
echo '{"test_prompt":"..."}' | python tests/uat/run_uat.py --branch master --agents gemini

Cost Awareness

Each scenario invocation costs API credits (one per agent per phase). Design scenarios efficiently:

• Combine related checks in a single test_prompt when possible
• Only use setup/teardown when the test needs specific state
• Start with one agent, expand to both only when cross-agent comparison matters

Handling Arguments

When /bat-adhoc is invoked with arguments:

If arguments contain a scenario description, generate the JSON scenario and run it:

/bat-adhoc test automation create with sunrise trigger then modify to sunset

→ Generate appropriate scenario JSON and execute

If --help or no arguments, show this help text.

Otherwise, treat $ARGUMENTS as instructions for what to test and design+run the scenario accordingly.

Full Documentation

For complete CLI reference and output format, see tests/uat/README.md.