OpenParable
Skip to content
← Back to skills

Bench Rule Enable

Testing & Quality
bench-rule-enable

Bench Rule Enable is an Agent Skill for testing & quality published in the markuplint/markuplint repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Enable an existing or newly-added markuplint rule on the nu-validator bench by editing tests/external/bench/config.ts. Covers the flat-rule case (rules: { '<name>': true }) and the severity override case (rules: { '<name>': { severity: 'error' } }) that bench-virtual-rule does not. Use when a rule exists in the registry but the relevant nu fixtures stay nu-only because the rule is not enabled in the bench config, or when a rule's default severity is warning but the spec text it mirrors is a MUST / MUST NOT. Trigger keywords: enable bench rule, bench config flat rule, severity override bench, rule not flagging on bench, warning vs error in bench, escalate severity bench, wai-aria-implicit-props bench, bench coverage missing rule.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/markuplint/markuplint/blob/HEAD/.claude/skills/bench-rule-enable/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/bench-rule-enable, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Bench Rule Enable Skill

tests/external/bench/config.ts curates rules explicitly (no extends: ['markuplint:html-standard']). A rule that exists in the markuplint registry will not affect bench verdicts until it is added to that file. This skill covers the two non-virtual-rule patterns:

  • Flat enable: adding '<rule-name>': true for a rule whose default behaviour matches the bench's spec-conformance lens.
  • Severity override: adding '<rule-name>': { severity: 'error' } for a rule whose default severity is 'warning' but whose underlying assertion is a MUST / MUST NOT in the spec text.

For preset-defined virtual rules (nodeRules[] selectors), use bench-virtual-rule instead.

When to add a rule to the bench

The bench is the spec-conformance comparison surface. A rule belongs in bench/config.ts only when its assertions map onto a nu-validator capability. Use this checklist:

  1. The rule is registered in packages/@markuplint/rules/src/index.ts.
  2. The rule reports diagnostics with severity 'error' (or you intend to override severity — see the next section).
  3. Either:
    • nu-validator currently fires on the same markup pattern, OR
    • the spec the rule cites contains MUST / MUST NOT language and the discrepancy in nu's coverage is documented in bench-triage's audit log.

If both nu and the spec are silent, do not enable the rule in the bench — it would inflate ml-only without producing a spec-aligned signal.

Pattern 1: flat enable

Most new conformance rules ship as flat enable: add '<rule-name>': true to rules in tests/external/bench/config.ts (entries kept alphabetical). After editing:

yarn build --scope @markuplint/rules
yarn bench:update --target markuplint --filter '<bench fixture path>'

Expect the targeted fixtures to flip from nu-only to match-error. If a fixture stays nu-only, the rule is not matching that specific HTML (re-read the rule and the fixture).

Pattern 2: severity override

A rule whose default meta.defaultSeverity is 'warning' does not contribute to bench verdicts (see compare.ts#judgeMlState — only errors flip ml state). Override for the bench-only run by adding '<rule-name>': { severity: 'error' } to bench/config.ts.

The inline comment on the override must include:

  • The spec citation that justifies treating the rule as error
  • A reason why the user-facing default stays at 'warning' (so the next maintainer does not "harmonise" by raising the user-facing default unconditionally)

Side-effect check

Severity overrides have ripple effects: fixtures that previously were match-clean (both clean) may flip to ml-only (ml strict, nu lax). This is not a regression when:

  • The spec the rule cites is a MUST / MUST NOT, and
  • The fixture's filename includes -haswarn (nu fires only a warning, not an error)

Confirm by inspecting tests/external/snapshots/diff/markuplint-only.json after the bench run: filter entries[] to those whose ruleIds include the rule.

Each affected fixture should be a legitimate "ml strict, nu lax" case per the bench-triage SKILL ml-only readings. Anything else is a real over-detection — back out the severity override.

Verification

yarn bench:update --target markuplint
yarn bench:compare

Compare tests/external/snapshots/diff/summary.md:

  • Per-category match-error count for the rule's domain should rise
  • nu-only count should fall by the same amount (within nu-validator parallel-load flicker)
  • ml-only delta should match the side-effect analysis above

Pin against --concurrency 1 before filing for a stable verdict.