OpenParable
Skip to content
← Back to skills

Archestra Dev Bench Analysis

Testing & Quality
archestra-dev-bench-analysis

Archestra Dev Bench Analysis is an Agent Skill for testing & quality published in the archestra-ai/archestra repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Map-reduce a finished archestra-bench run into a Tier-1/Tier-2 improvement report using Claude subagents (same analysis as the Rust analyzer, no API key).

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/archestra-ai/archestra/blob/HEAD/.claude/skills/archestra-dev-bench-analysis/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/archestra-dev-bench-analysis, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Archestra bench trajectory analysis

Map-reduce a finished archestra-bench run into a recommendations report, using Claude subagents for the judgment. The deterministic half (render + metrics + manifest) is done by the Rust archestra-bench prepare subcommand, so this skill reuses the analyzer's exact rendering, metrics, and ordering — it does not re-implement them. Mirrors ai-labs/analyzer (map = per-rollout triage; reduce = repo-grounded Tier-1/Tier-2 report); see ai-labs/analyzer/README.md.

The subagent fan-out runs through the native Workflow tool — two scripts under this skill's workflows/ directory drive the map and crawl phases, so the orchestrator never hand-batches Agent calls and the per-rollout triages never flow through its context. Calling those scripts here is your explicit opt-in to Workflow. The exact map/reduce prompt text lives in reference/prompts.md (this skill's directory) — bin/prepare.sh extracts the MAP block automatically; you still read the REDUCE sections verbatim in step 4. Do not paraphrase those prompts.

<SKILL_DIR> below is this skill's absolute directory (the one containing this file).

1. Prepare (deterministic: dir resolution + Rust prepare + arg shaping)

Run the helper (from anywhere — it derives the repo root). Pass a run dir, or omit it to pick the newest under ai-labs/experiments/:

<SKILL_DIR>/bin/prepare.sh "$ARGUMENTS"

It resolves the run dir to an absolute path, runs archestra-bench prepare (failures-first manifest; fail-fast — if it exits non-zero, surface the printed path:line error and stop), and writes under <RUN_DIR>/_prep_claude/: manifest.json, metrics.md, order.tsv (idx<TAB>id<TAB>outcome, manifest order), one pre-rendered triage prompt per rollout under prompts/<NN>.txt, and map-args.json (small, ready-to-pass args for the map workflow). It prints a KEY=value summary — capture RUN_DIR, TS, TRIAGE_DIR, MAP_ARGS, ANALYSES_DOC, REPORT_DOC — then the metrics block. State which RUN_DIR it chose.

2. Map — one triage workflow

Read the MAP_ARGS file (map-args.json) and pass its JSON as args to the Workflow tool with scriptPath: <SKILL_DIR>/workflows/map.mjs. The file is small by design — triageDir, promptsDir, and the rollouts array ({idx,id}, manifest order); the filled triage prompts live on disk under promptsDir, so the bulk never flows through your context. The workflow fans out one Sonnet triage agent per rollout (auto-batched at the concurrency cap — no manual 8-at-a-time loop); each reads its pre-rendered prompt (<promptsDir>/<NN>.txt) and the trajectory it points to, then Writes its judgment — a single JSON object (rubric grades, reward-hacking flag, observations) — to <TRIAGE_DIR>/<NN>.json (zero-padded idx). It returns {written, total}; if written < total, step 3's validator will name the missing indices. Sonnet is deliberate — triage is a cheap bounded read; reserve the stronger model for the reduce synthesis.

3. Validate + assemble (deterministic, in this loop)

Run the renderer — it validates every <TRIAGE_DIR>/<NN>.json against order.tsv, stamps rollout/outcome from order.tsv (never from model output), writes the rubrics JSONL (<RUN_DIR>/trajectory_rubrics_claude_<TS>.jsonl) and the analyses doc (truncation parity with the Rust analyzer included), and prints both paths as KEY=value:

node <SKILL_DIR>/bin/render-triage.mjs <RUN_DIR> <TS>

If it exits non-zero, it lists the invalid/missing indices (and any extra/stale files) on stderr: delete the extra files, re-run the map workflow with a rollouts subset containing only those indices — filter map-args.json with jq, e.g. jq '.rollouts |= map(select(.idx == 3 or .idx == 7))' <MAP_ARGS> — then re-run render-triage.mjs. Do this before the reduce step — a reduce failure must never discard the map work.

4. Reduce — repo-grounded report

Read the analyses doc (ANALYSES_DOC). Adopt the REDUCE system guidance from reference/prompts.md as your framing, then carry out the REDUCE task message (fill {ANALYSES_DOC_PATH} with ANALYSES_DOC, {RUN_DIR} absolute, {BACKEND_LOG_PATHS} with <RUN_DIR>/*.backend.log).

Ground every finding in file:line across platform/ (Tier 1) and ai-labs/ (Tier 2) by fanning the crawlers out through the crawl workflow: derive one issue/subsystem per cluster from the analyses doc, then call the Workflow tool with scriptPath: <SKILL_DIR>/workflows/crawl.mjs and args:

{
  "repoRoot": "<repo root absolute>",
  "crawlerSystem": "<the verbatim REDUCE crawler system prompt from reference/prompts.md>",
  "issues": [ { "label": "run_command-target", "prompt": "<one issue to investigate>" }, ... ]
}

It returns [{ label, evidence }] (.filter(Boolean) it — a hard-failing crawler yields a null) — the grounding you synthesize the report from. Before citing a surprising map claim, open the rollout's raw trajectory.md and confirm it.

Hard rule: the backend-log files (<RUN_DIR>/*.backend.log, e.g. basic.backend.log) are ~tens of MB. Never Read/cat them (yours or a subagent's) — only capped grep, e.g. grep -n -m 50 -F '<pattern>' <RUN_DIR>/basic.backend.log.

Write the report to REPORT_DOC (<RUN_DIR>/trajectory_analysis_claude_<TS>.md).

5. Report

Tell the user the output paths (analyses doc, rubrics JSONL, report) and a one-line headline (overall pass rate + the top Tier-1 finding).