OpenParable
Skip to content
← Back to skills

Regen Shelltests

Testing & Quality
regen-shelltests

Regen Shelltests is an Agent Skill for testing & quality published in the simonmichael/hledger repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Regenerate expected output in failing shelltestrunner v3 fixtures (`hledger/test/**/*.test`) after a hledger output-format change (alignment, column shifts, rendering tweaks). Use when many `.test` fixtures are stale because of a deliberate, audited rendering change.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/simonmichael/hledger/blob/HEAD/.claude/skills/regen-shelltests/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/regen-shelltests, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Regenerating shelltestrunner fixtures

When a change to hledger's output rendering (alignment, padding, column positions) breaks many fixtures in hledger/test/, use tools/regen-shelltests.py to bulk-update them.

The script's default mode is whitespace-only: it overwrites a test's expected stdout with the actual output only when the difference is purely whitespace (column shifts). Anything else — added/removed lines, changed numbers, different commodities — is left for human review. This makes it safe to run on a batch of failing tests without losing in-flight content edits.

When to use this skill

The user is dealing with many failing functional tests after a deliberate rendering change, and wants the column shifts auto-applied while preserving any other content for hand-review.

Do NOT use this skill if the user wants to regenerate all output blindly without auditing — that's a sign the underlying change isn't well understood, and a hand-review of a few representative failures is the right next step.

Workflow

  1. Build hledger first — the script invokes stack exec -- which hledger to find the binary and prepends its dir to PATH so test wrapper scripts (e.g. csvtest.sh) pick up the just-built one.

    stack build hledger
    
  2. Identify failing test files (not individual tests — the script processes whole files):

    stack exec -- shelltest --execdir --exclude=/_ hledger/test/ \
      -x hledger/test/perf.test \
      -x ledger-compat/ledger-baseline -x ledger-compat/ledger-regress -x ledger-compat/ledger-extra \
      -j8 2>&1 \
      | grep "Failed\]" \
      | sed 's/^:\(.*\):[0-9]*: \[Failed\]$/\1/' \
      | sort -u > /tmp/failing-files.txt
    
  3. Run the regen on just those files:

    cat /tmp/failing-files.txt | xargs tools/regen-shelltests.py
    

    Output looks like:

    hledger/test/csv.test: 63 tests, 3 updated, 7 skipped
    ...
    TOTAL: 102 tests, 9 updated, 17 skipped
    

    "Updated" = expected stdout overwritten because the diff was whitespace-only. "Skipped" = left as-is for one of these reasons (the script logs each):

    • Test has a stderr expectation (>2 … or >>>2 …).
    • No exit marker (malformed or v2-format).
    • Exit code mismatched (real failure, not just formatting).
    • Diff was more than whitespace.
  4. Re-run the tests to confirm the regen reduced failures:

    just functest --hide
    
  5. Hand-fix the remainder. Common remaining cases:

    • Stderr-regex tests with embedded posting renders (errors/, journal/parse-errors.test, etc.) — adjust the spaces/carets in the regex by hand.
    • Tests without exit markers — add >= or >= 0 if the format is malformed.
    • Real content changes — investigate; these are not just-formatting issues.
  6. git diff and review before committing. The whitespace-only safeguard is good but not infallible.

Flags

  • --all — bypass the whitespace-only check and overwrite whenever the run produced the expected exit code. Use only after you've audited a few failures and confirmed the actual output is correct in a non-whitespace way.

Limitations

  • Doesn't handle stderr regenerate. Stderr regex blocks (>2 /…/ and >>>2 /…/) are skipped because regex regen is fiddly (escaping, multi-line patterns, regex-tdfa size limits).
  • Doesn't handle v2-format input blocks (<<< / >>> / >>>=). Files using those are passed through verbatim.
  • Doesn't update doctests in Haskell sources (run just doctest separately, then hand-update).
  • Regenerates in place. Always work on a clean tree so git diff is meaningful for review.

Anti-patterns

  • Regen with --all on a fresh transcript without inspection. This is the same as accepting whatever hledger emits today as correct, including bugs. Always run with the default (whitespace-only) first; only escalate to --all for tests where you've manually verified the new output.
  • Regen across the whole hledger/test/ directory. The whitespace-only mode protects against this somewhat, but it's still safer to feed only the failing files — that way you'll notice if a passing test mysteriously changes.
  • Skipping the rebuild. If stack build hledger is stale, the regen reflects yesterday's binary, not your latest code. Always build first.