writing-ratchet-tests

Writing Ratchet Tests

This skill provides guidelines for writing ratchet tests that prevent accumulation of code anti-patterns in a project.

What are Ratchet Tests?

Ratchet tests are a testing pattern that:

• Track the current count of a specific anti-pattern in the codebase
• Prevent that count from increasing (using inline-snapshot)
• Allow the count to decrease (improvement is always allowed)
• Provide clear, actionable feedback when violations increase

Common use cases:

• TODO comments
• Inline imports
• Use of eval() or exec()
• Broad exception handling (bare except, except Exception)
• Any other code pattern you want to gradually eliminate

Architecture

The ratchet system has three layers:

1. Rule definitions in libs/imbue_common/imbue/imbue_common/ratchet_testing/common_ratchets.py -- RegexRatchetRule and RatchetRuleInfo objects
2. Wrapper functions in libs/imbue_common/imbue/imbue_common/ratchet_testing/standard_ratchet_checks.py -- one function per ratchet rule
3. Test functions in each project's test_ratchets.py -- call the wrapper with a snapshot count

All projects must have the same set of test functions (enforced by test_meta_ratchets.py).

Adding a New Common Ratchet

Step 1: Define the Rule

Add a RegexRatchetRule to common_ratchets.py:

PREVENT_MY_PATTERN = RegexRatchetRule(
    rule_name="my pattern usages",
    rule_description="Explain why this pattern is problematic and what to do instead",
    pattern_string=r"my_regex_pattern",
    is_multiline=False,  # set True if using ^ or $ anchors
)

Or a RatchetRuleInfo for AST-based checks (add the detection function to ratchets.py).

Step 2: Add a Wrapper Function

Add to standard_ratchet_checks.py (the single source of truth for which ratchets exist):

def check_my_pattern(source_dir: Path, max_count: int) -> None:
    assert_ratchet(PREVENT_MY_PATTERN, source_dir, max_count)

Remember to import the rule at the top of the file. The function name determines the test name: check_foo becomes test_prevent_foo.

Step 3: Sync and Set Counts

uv run python scripts/sync_common_ratchets.py
uv run pytest --inline-snapshot=update -k test_ratchets

The sync script reads standard_ratchet_checks.py, generates test_prevent_my_pattern in all 26 test_ratchets.py files with snapshot(0), then the pytest command sets the actual violation counts per project.

Adding a Project-Specific Ratchet

Not all ratchets belong in every project. If a ratchet only applies to one project (e.g., a project-specific API convention), add it to a separate file -- NOT to test_ratchets.py (which must define the same test function names across all projects, enforced by test_meta_ratchets.py).

Create a project-specific ratchet test file (e.g., test_project_ratchets.py) using the core API directly:

from pathlib import Path

from inline_snapshot import snapshot

from imbue.imbue_common.ratchet_testing.core import FileExtension
from imbue.imbue_common.ratchet_testing.core import RegexPattern
from imbue.imbue_common.ratchet_testing.core import check_regex_ratchet
from imbue.imbue_common.ratchet_testing.core import format_ratchet_failure_message

_DIR = Path(__file__).parent.parent.parent


def test_prevent_my_project_pattern() -> None:
    pattern = RegexPattern(r"my_regex_pattern", multiline=False)
    chunks = check_regex_ratchet(_DIR, FileExtension(".py"), pattern)
    assert len(chunks) <= snapshot(), format_ratchet_failure_message(
        rule_name="my project pattern",
        rule_description="Why this is problematic and what to do instead",
        chunks=chunks,
    )

Run with --inline-snapshot=create to set the initial count, then verify it passes normally.

Important Rules

• Never add per-project code quality ratchets to test_meta_ratchets.py. Meta ratchets are for repo-wide structural checks only.
• If a ratchet applies to all projects, use the common ratchet workflow (sync script). If it only applies to one project, use a separate test file with the core API.
• Keep test function names descriptive: test_prevent_<anti_pattern>
• Provide clear rule_description that explains WHY the pattern is bad and WHAT to do instead
• Never blindly update snapshots -- investigate why a count increased

Troubleshooting

Pattern not matching expected violations:

• Check if you need multiline=True for patterns using ^ or $
• Verify the regex is correct using a regex tester
• Check that the file extension is correct
• Ensure violations are in git-tracked files (git blame only works on committed code)

Test fails after running:

• This is expected if the current count is higher than the snapshot
• Always fix the violations if a ratchet test fails--it's because you messed something up. NEVER run with --inline-snapshot=fix
• Never just blindly update snapshots - investigate why the count increased

Snapshot shows 0 but violations exist:

• The regex pattern might be incorrect
• Try running with multiline=True if using ^ or $ anchors