Routing Calibration Loop

Routing Calibration Loop

Trigger

• Use when a signal, projection, decision, or maintained routing example needs to be checked against a live router apiserver
• Use when a routing failure must be classified as a bad probe, bad routing policy, or bad validator rule instead of blindly patching the profile
• Use when a maintainer wants the loop eval -> update -> validate -> deploy -> eval to be run with versioned evidence

Required Surfaces

• harness_docs

Conditional Surfaces

• harness_exec
• router_service_platform
• router_config_contract
• signal_runtime
• decision_logic
• algorithm_selection
• dsl_crd
• docs_examples

Stop Conditions

• No live router base URL is available and no local replacement environment has been chosen
• No probe manifest exists and the task cannot safely infer executable probes from maintained examples
• A deploy would change remote runtime state without capturing the current version or without a rollback path
• Local validation fails for reasons that are not yet understood or recorded

Workflow

1. Start from executable probes, not prose examples.
   • Prefer a machine-readable manifest. deploy/recipes/balance.probes.yaml is the default maintained example, not the only supported target.
   • The manifest should stay profile-generic: point to any owned routing YAML / DSL pair through routing_assets, and group probes by decision with multiple variants when robustness matters.
   • Treat each probe as both a test case and a specification fragment.
2. Baseline the live router before editing policy.
   • Use tools/agent/scripts/router_calibration_loop.py to snapshot /config/router and /config/router/versions, then run /api/v1/eval across the probe suite.
   • Record which decision actually fired, which signals matched, and which signals were expected but absent.
3. Classify every failure under one of three buckets before changing anything.
   • query_quality: the prompt is not a robust representative of the intended route.
   • routing_design: the signal / projection / decision design is too broad, too narrow, or too brittle.
   • validator_quality: the runtime behavior is reasonable but static validation is over-reporting or under-reporting.
4. Edit the canonical authoring surface locally.
   • For maintained routing, edit the owned YAML / DSL asset pair instead of patching only the live server.
   • Do not add narrow trigger-phrase hacks just to pass one probe.
5. Run local validation before deploying.
   • Use the runner's run or validate path to execute sr-dsl validate against the DSL source, or against a YAML file through decompile-then-validate.
   • Prefer manifest-owned assets as defaults, but allow explicit YAML / DSL overrides for any other routing profile.
   • Keep validation output with the loop artifacts so validator behavior can be reviewed alongside runtime eval output.
6. Deploy durably and re-evaluate.
   • Use PUT /config/router for versioned full-document replacement so the live router exactly matches the canonical YAML being calibrated.
   • After every config update, wait for GET /ready to return ready=true before trusting eval results. Do not treat a successful update response as proof that router initialization has finished.
   • Re-run the same probe suite after deploy and compare before / after success rate and per-probe traces.
7. Close the loop with structured reflection.
   • 0. Query quality: Is the probe semantically representative, or is it a brittle phrase trigger?
   • 1. Routing design: Are the signal, projection, and decision boundaries robust, or merely sufficient for this probe set?
   • 2. Validator quality: Do warnings or failures reflect real ambiguity, or missing static semantics?
8. If a durable architecture gap remains, update the indexed debt entry instead of leaving the mismatch only in chat or the report.

Gotchas

• The calibration loop now deploys with PUT /config/router, so the calibrated YAML must be a complete router document, not a partial merge fragment.
• Do not declare success just because one crafted query passes. Probe quality is part of the task; decision-level robustness should be checked with multiple variants, not just one trigger phrase.
• If runtime eval looks correct and validation still looks wrong, assume validator semantics may need work rather than forcing a worse route design.
• If deploy succeeds but success rate regresses, capture the returned version and use the versions endpoint before continuing.

Must Read

• AGENTS.md
• deploy/amd/README.md
• deploy/recipes/balance.probes.yaml
• tools/agent/scripts/router_calibration_loop.py

Standard Commands

• python3 tools/agent/scripts/router_calibration_loop.py eval --router-url http://<router-host>:8080 --probes <profile>.probes.yaml
• python3 tools/agent/scripts/router_calibration_loop.py run --router-url http://<router-host>:8080 --probes <profile>.probes.yaml
• python3 tools/agent/scripts/router_calibration_loop.py run --router-url http://<router-host>:8080 --probes <profile>.probes.yaml --yaml <routing>.yaml --dsl <routing>.dsl
• python3 tools/agent/scripts/router_calibration_loop.py deploy --router-url http://<router-host>:8080 --yaml <routing>.yaml --dsl <routing>.dsl --ready-timeout 300
• make agent-report ENV=amd CHANGED_FILES="deploy/recipes/balance.yaml,deploy/recipes/balance.dsl,deploy/amd/README.md"
• make agent-ci-gate CHANGED_FILES="tools/agent/skills/maintainer/routing-calibration/SKILL.md,tools/agent/scripts/router_calibration_loop.py,deploy/recipes/balance.probes.yaml"

Acceptance

• Each calibration round produces a probe report with before / after outcomes, live decision traces, and the captured deploy version when a deploy occurs
• Failures are explicitly reviewed under query quality, routing design, and validator quality instead of being patched blindly
• Maintained routing changes are validated locally before deploy and re-evaluated on the live endpoint after deploy
• The loop leaves behind executable probes or maintained examples that are stronger than the ones it started with, ideally by improving decision-level variant coverage instead of adding single-example hacks