OpenParable
Skip to content
← Back to skills

Test Confidence

Testing & Quality
test-confidence

Test Confidence is an Agent Skill for testing & quality published in the antiwork/gumroad repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

AI-driven test execution. Opus decides what to run and how confident to be, based on your diff.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/antiwork/gumroad/blob/HEAD/.agents/skills/test-confidence/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/test-confidence, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Test Confidence

Run bin/test-confidence to have Opus 4.7 analyze your diff, decide the risk level, plan which tests to run and in what order, and set confidence milestones. The AI decides the shape of the curve based on this specific diff.

Usage

bin/test-confidence            # Run to 99%, stop. Skips past pre-existing failures.
bin/test-confidence --full     # Run to 100%
bin/test-confidence --strict   # Halt on any failure, including pre-existing

ANTHROPIC_API_KEY is auto-sourced from .env if not exported.

If $ARGUMENTS is provided, pass it through: bin/test-confidence $ARGUMENTS

How it works

  1. Finds changed files (branch diff vs main for PR branches, local diff on main)
  2. Hashes the diff and checks tmp/test-confidence/ for a cached plan; reuses if found
  3. Otherwise sends the diff + spec tree + touched directories to Opus 4.7 in one call
  4. Opus returns a plan: risk level, ordered test list, confidence milestones
  5. Script executes the plan, showing yellow progress bar toward 99%
  6. At 99%, bar turns green. Safe to commit.
  7. With --full, continues running remaining tests toward 100%

Pre-existing failure detection

When a spec fails, the script applies a two-step check:

  1. Path heuristic — does the failing spec file or its source counterpart (e.g., app/models/user.rb for spec/models/user_spec.rb) appear in the diff?
    • In the diff → real regression. Halt immediately.
    • Not in the diff → suspect pre-existing. Verify on merge-base.
  2. Merge-base verify — re-run the failing rspec examples in a temporary worktree at the branch's merge-base with main.
    • Still fails on merge-base → confirmed pre-existing. Continue.
    • Passes on merge-base → cross-file regression caught. Halt.

The verify step is skipped (heuristic verdict trusted) when the diff includes a db/migrate/*.rb (DB schema drift) or Gemfile.lock (bundler drift), or when no merge-base is available. --strict skips both checks and halts on every failure.

Cost profile: zero overhead in the common no-failure case; verify only runs on heuristic-flagged "pre-existing" failures. Catches the dangerous direction (silent regression skipped) while keeping false-alarm investigation costs bounded.

The key insight: Opus decides ad hoc how many tests are needed for each confidence level. A comment-only change might need 2 tests for 99%. A payment model refactor might need 100.

When to use

Run this before every commit. It replaces manually picking which specs to run.

$ARGUMENTS