coverage-mapping

Coverage Mapping Strategy

Map the evaluation landscape for a domain to identify which capabilities are well-tested, which are undertested, and which have no evaluation coverage at all. Produces a capability taxonomy with benchmark coverage annotations.

Purpose

Build a comprehensive map of "what we can and cannot measure" for a given AI capability domain. Identify white spaces where important capabilities lack rigorous evaluation, and redundancies where multiple benchmarks test the same narrow skill.

Budget

State Ledger

<HARD-GATE>
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| Benchmarks mapped | 0 | 20 | PENDING |
| Capability taxonomy nodes | 0 | 30 | PENDING |
| Papers fetched | 0 | 30 | PENDING |
| Papers read | 0 | 20 | PENDING |
| Web searches | 0 | 50 | PENDING |
| Coverage annotations complete | 0 | 20 | PENDING |
| White spaces identified | 0 | 5 | PENDING |
| Redundancy clusters found | 0 | 3 | PENDING |
</HARD-GATE>

Cannot exit until 80% of all targets met.

Available Tactics

• score-trajectory-analysis — Understand maturity level of each benchmark

Available SOPs

• benchmark-inventory — Catalog all benchmarks in domain
• capability-taxonomy-mapping — Build hierarchical capability taxonomy
• metric-decomposition — Understand what each benchmark actually measures
• benchmark-synthesis — Produce coverage map report

Execution Guidance

1. Domain Scoping: Define the capability domain and its boundaries
2. Taxonomy Construction: a. Run capability-taxonomy-mapping to build hierarchical capability tree b. Use papers and web searches to refine taxonomy with community consensus
3. Benchmark Inventory: a. Run benchmark-inventory to collect all known benchmarks in domain b. For each benchmark, run metric-decomposition to identify tested capabilities
4. Coverage Annotation: a. Map each benchmark to taxonomy nodes it covers b. Identify coverage density per node (over-tested vs under-tested) c. Mark white spaces (zero coverage nodes)
5. Redundancy Analysis: Cluster benchmarks that test identical capabilities
6. Maturity Assessment: Run score-trajectory-analysis on key benchmarks to assess evaluation maturity
7. Synthesis: Produce coverage map with gap prioritization

Output Format

coverage_map:
  domain: string
  taxonomy:
    - node: string
      level: int
      children: list
      coverage_status: well-covered|partial|minimal|none
      benchmarks: list[string]
  white_spaces:
    - capability: string
      importance: high|medium|low
      reason_untested: string
      proposed_evaluation: string
  redundancy_clusters:
    - capability: string
      benchmarks: list[string]
      differentiation: string
  coverage_statistics:
    total_capabilities: int
    well_covered: int
    partial: int
    minimal: int
    none: int
    coverage_ratio: float

Available Tactics

Optional, no fixed order; the final leaf is always a sop.

Available SOPs

Optional, no fixed order; the final leaf is always a sop.