run-napkin-math-pipeline

Run the Napkin-Math Pipeline

Overview

End-to-end orchestrator for the pipeline documented in experiments/napkin_math/README.md. Decides which stages to run based on what is already present in the target output directory, then walks forward through the remaining stages one at a time.

PlanExe source reports live under /Users/neoneye/git/PlanExe-web/ (e.g. /Users/neoneye/git/PlanExe-web/20251114_paperclip_automation/). Outputs land under experiments/napkin_math/output/<version>/<plan-slug>/.

What the orchestrator does and does not do

The orchestrator is dispatch-only. It looks at filenames in the target directory, decides which stage is missing, and delegates the work to the appropriate sibling skill (via a subagent) or Python runner. It does not read the content of large input files into its own context.

Do not read into orchestrator context:

• extract_parameters_input.md (the digest — typically ~25K tokens)
• compress_*.md or compress_*_raw.json files
• The raw PlanExe HTML report
• The full parameters.json, bounds.json, scenarios.json, montecarlo.json (except quick jq/Python extraction of a single field)

OK to read into orchestrator context:

• Directory listings (filenames only)
• validation.json exit status (via Python one-liner, not full file)
• montecarlo.json per-threshold probability values (via Python one-liner)
• Sibling skill SKILL.md / system-prompt.txt only when updating the skill itself

Pulling a digest into orchestrator context defeats the architecture. Each stage's sibling skill knows how to read its own inputs; the orchestrator's job is to invoke that skill, not to do its work.

Per-stage delegation

For each missing stage, dispatch as follows. Pass the agent the file paths and a one-line task; do not paste file contents into the prompt.

After each agent returns, verify the output file exists and is non-empty, then move on. If validation fails after Stage 1, dispatch a repair agent with the validation report — do not pull the parameters into context yourself.

Why resume mode exists

Resume is not a convenience. It is the experimental control for this pipeline.

Several stages — digest preparation, parameter extraction, bounds, calculations, scenarios — are LLM-driven and therefore non-deterministic. If every run regenerated them from scratch, every downstream change would be confounded with upstream sampling noise, and the user could not tell whether a different assessment.md came from a skill/prompt tweak or just from a different roll of the LLM.

The intended workflow is:

1. On the first run for a plan, populate the directory once. The extract_parameters_input.md digest (Stage 0) becomes the pinned starting point.
2. On subsequent runs — typically after editing a skill prompt, a validator rule, or a runner — the user deletes only the artifacts downstream of the change, and resumes from the highest-numbered stage whose output is still present.
3. Because the input to the first re-run stage is byte-identical to the prior run, any downstream difference can be attributed to the tweak under test.

The corollary is strict: never re-run a stage whose output is already on disk. Doing so silently changes the input that every later stage sees, and destroys the comparability the user is trying to preserve. If the user wants a stage re-run, they will delete its output file first; absence of the file is the signal to run, presence is the signal to leave alone.

The two cardinal rules

1. Never copy a pipeline artifact from a different output directory into the target directory. Skill prompts, validator rules, and runner logic evolve between versions; copying output/<vN-1>/<plan>/<file> into output/<vN>/<plan>/ silently embeds the old rules into the new run. If at any point you are about to type cp /…/v{N-1}/<plan>/<file> $D/, stop and run the stage that produces <file> instead.

2. Never re-run a stage whose output is already on disk. The user's A/B comparability depends on the input to each stage being byte-identical to the prior run, up to the first deleted artifact. Re-running a present stage (especially Stage 0, the LLM-driven digest) re-rolls the dice on its input distribution and confounds every downstream comparison. The only file presence the orchestrator changes is by adding missing files, never by replacing present ones.

Together: file present ⇒ leave alone; file absent ⇒ produce it from this stage's runner (never by copy).

Pipeline file map

Each stage produces exactly one or two files in the target directory. A stage is "done" iff its output file(s) exist in the target dir.

montecarlo_settings.json is a tiny JSON file (n_runs, seed, distribution_default, thresholds) — write it from the threshold list in recommended_first_calculations, with >= 0 operator for every margin/surplus/buffer output. Standard values: n_runs: 10000, seed: 12345, distribution_default: "triangular".

Inputs the orchestrator needs

Two scenarios:

(a) Fresh start from a PlanExe-web report. User points at /Users/neoneye/git/PlanExe-web/<date>_<slug>/ and a target version. Create output/<version>/<slug>/ if it doesn't exist. Run stage 0 first.

(b) Resume from a partially populated output directory. User points at output/<version>/<slug>/. List the dir, classify which stages are done, run forward from the first missing stage.

If the user did not name a version, ask. Do not guess from sibling directories (last-mtime, etc.).

Workflow

1. Identify the target directory. Absolute path under experiments/napkin_math/output/<version>/<plan-slug>/.
2. List its contents. Decide per stage whether the output file is present.
3. Find the first missing stage. Run it. After it completes, re-check and continue with the next missing stage. Do not batch multiple stages together until you have verified the previous one landed.
4. For LLM-driven stages, invoke the corresponding sibling skill. Each sibling skill's system-prompt.txt is authoritative — read it and apply its rules. The orchestrator does not paraphrase or short-circuit those rules.
5. For Python stages, invoke the script. See the commands below.
6. Re-validate after each non-deterministic stage. Specifically: after parameter extraction, run validate-parameters and confirm valid: true before moving on. If valid: false, hand-fix the reported violations and re-validate; do not proceed with invalid parameters.
7. Final summary. When assessment.md exists and Monte Carlo completed with zero warnings, report the worst-gate verdict, the five pass probabilities, and the overall band.

Commands (Python stages)

Always invoke Python 3.11 explicitly (the runner uses NumPy + llama_index that resolve under that interpreter):

PY=/opt/homebrew/bin/python3.11
NM=/Users/neoneye/git/PlanExeGroup/PlanExe/experiments/napkin_math

Stage 0 — digest preparation (creates the 8 compress files + the digest):

$PY $NM/prepare_extract_input.py \
  --planexe-dir /Users/neoneye/git/PlanExe-web/<date>_<slug> \
  --output-dir  $NM/output/<version>/<plan-slug>

Stage 2 — validation:

$PY $NM/validate_parameters.py \
  --parameters $D/parameters.json \
  --output     $D/validation.json

Stage 7 — Monte Carlo:

$PY $NM/run_monte_carlo.py \
  --parameters   $D/parameters.json \
  --bounds       $D/bounds.json \
  --calculations $D/calculations.py \
  --settings     $D/montecarlo_settings.json \
  --output       $D/montecarlo.json

Stage 8 — assessment rendering:

$PY $NM/summarize_assessment.py \
  --parameters   $D/parameters.json \
  --bounds       $D/bounds.json \
  --scenarios    $D/scenarios.json \
  --montecarlo   $D/montecarlo.json \
  --validation   $D/validation.json \
  --settings     $D/montecarlo_settings.json \
  --extract-input $D/extract_parameters_input.md \
  --calculations $D/calculations.py \
  --output       $D/assessment.md

After Stage 7, delete the __pycache__/ that the runner creates next to calculations.py:

rm -rf $D/__pycache__

LLM-driven stages — how to actually run them

For stages 1, 3, 4, 5: do not paste the upstream artifacts and improvise. Walk through the sibling skill's workflow:

• Stage 1 — extract-parameters-from-digest
  • Read $D/extract_parameters_input.md.
  • Read ../extract-parameters-from-digest/system-prompt.txt.
  • Produce parameters.json per the schema at the end of the system prompt. Apply every "Important", "Additional modelling rules", and "Formula and dependency rules" section.
• Stage 3 — generate-bounds
  • Read $D/parameters.json (must be valid).
  • Read ../generate-bounds/system-prompt.txt and SKILL.md.
  • Apply the actual-vs-commitment default: base centers at the source-named committed value unless a named Risk / Issue / Decision / premortem / expert-criticism passage forecasts a gap.
  • Skip threshold/target variables (ids ending in _threshold, _target, _ceiling, _floor, _limit) — they enter the simulation as their stated single value.
• Stage 4 — generate-calculations
  • One Python function per entry in recommended_first_calculations. Function name = output_name; arguments = depends_on. No hidden state; add divide-by-zero guards only where the formula divides.
• Stage 5 — run-scenarios
  • Build three input pools (low / base / high) by selecting the matching value from bounds.json for bounded variables, and the declared key_values[*].value for unbounded variables.
  • Run each recommended_first_calculations function against each pool.
  • Emit scenarios.json with inputs, outputs, comparison (low/base/high + spread_ratio + spread_absolute), and any warnings. Compute spread arithmetically; do not copy the structure from a prior run.

Resume-mode decision table

When the user points at a partially populated dir, decide by file presence only — never by sibling-directory comparison.

If an intermediate file is present but a later one is also present with stale numbers (e.g. bounds.json updated but montecarlo.json not regenerated), ask the user whether to keep or regenerate the later file. Do not silently overwrite.

Anti-patterns

Self-audit before final report

Before declaring the run complete, verify:

1. validation.json shows valid: true and 0 errors.

2. montecarlo.json has 0 warnings, or the warnings are understood and acknowledged in the summary.

3. Every threshold id in montecarlo_settings.json matches an output_name in parameters.json's recommended_first_calculations.

4. assessment.md exists and its ## Machine summary block lists overall_risk_band consistent with the worst per-gate pass probability under the band table:

   Pass rate	Band
   ≥ 0.80	Robust
   0.50 – 0.80	Marginal
   0.20 – 0.50	Fragile
   < 0.20	Critical

5. No __pycache__/ directory remains in the output dir.

If any check fails, fix it before reporting back. Reporting "done" with a known issue is worse than asking the user how to proceed.

Reference

• Pipeline overview: ../../../README.md (Stages 1–7 + design principles)
• Sibling skill prompts (authoritative for each stage): ../extract-parameters-from-digest/, ../generate-bounds/, ../generate-calculations/, ../run-scenarios/, ../monte-carlo/, ../summarize-assessment/, ../validate-parameters/
• Python runners: ../../../prepare_extract_input.py, ../../../validate_parameters.py, ../../../run_monte_carlo.py, ../../../summarize_assessment.py
• PlanExe source reports: /Users/neoneye/git/PlanExe-web/<date>_<slug>/
• Output convention: experiments/napkin_math/output/v<N>/<plan-slug>/