skill-grader

Skill Grader (Compliance Scorecard)

Purpose

This skill audits other skills for discoverability, instruction clarity, and token efficiency. It provides a formal "Compliance Scorecard" and an HTML audit report.

Compliance Scorecard Rubric

Operational Protocol

1. Simulated Consumption (The Dry Run)

• The grader reads the skill and its references/.
• It executes a test prompt based on the skill's purpose.
• It grades the result against the project's architecture standards.

2. The Grading Scale

• A+ (95-100%): Easily discovered, perfect test results, highly efficient.
• A (90-94%): Highly discoverable, minor polish needed.
• B (80-89%): Solid, but lacks "Gold Standard" polish.
• C (70-79%): Technically usable, but needs description/token cleanup.
• D (60-69%): Non-Critical Failure (often due to being hidden or failing non-critical optional items).
• F (< 60%): Critical Failure (Test suite failure, totally undiscoverable, or architectural violation).

3. Execution Workflow

1. Initial Scan: Audit SKILL.md and metadata.
2. Data Preparation:
   • The Grader MUST create/update memory/compliance_report.json using the template at assets/grading_report_template.json BEFORE executing any scripts.
   • Schema Verification: The generated JSON MUST match the schema required by the generation script, including mandatory fields: skill_name, audit_date, score (integer), grade, and the findings object.
3. Report Generation: Execute the mandatory generation script relative to the skill root to process this data: python3 scripts/generate_report.py ./
4. Final Review: Ensure memory/audit_report.html was generated by the script in the target skill's memory/ directory.

4. Mandatory Artifact Generation (Definition of Done)

• Automatic Directory Creation: If the target skill's memory/ directory does not exist, the Grader MUST create it before artifact generation.
• Persistent Reporting: Every execution of this skill MUST conclude by creating/updating:
  1. memory/compliance_report.json in the target skill directory.
  2. memory/audit_report.html in the target skill directory.
• Automatic Failure on Missing Artifacts: If these files are not created/updated at the end of the execution, the audit is considered incomplete and failed.
• MANDATORY TOOLING: The execution of python3 scripts/generate_report.py ./ is the ONLY way to create memory/audit_report.html. The agent is strictly forbidden from manually generating this file.
• Atomic Commitment (NON-NEGOTIABLE): The agent is STRICTLY PROHIBITED from reporting a "Final Grade" in the chat UNLESS the memory/compliance_report.json and memory/audit_report.html have been successfully created/updated on the filesystem via the mandatory script.
• Verification: Before providing a grade, the agent MUST verify the existence of both required files on the filesystem.

Strict Compliance

• All audits MUST be reported in the target skill's memory/ directory.
• Critical failures (Architectural/DI violations) trigger an automatic F.
• For programmatic consistency, all reports MUST follow the templates located in the assets/ directory:
  • audit_report.html MUST use assets/audit_report_template.html
  • compliance_report.json MUST use assets/grading_report_template.json

Deterministic Coaching (Suggestions for Improvement)

To ensure suggestions are thorough, deterministic, and project-agnostic, the Grader MUST evaluate the skill against these four "Coaching Pillars":

1. The "North Star" Alignment:

   • Check: Does the skill explicitly link to or reference the project's master architectural guides (e.g., AGENTS.md, architecture.md)?
   • Suggestion Pattern: "Strengthen architectural coupling by adding a 'Standards Reference' section pointing to [Missing File]."

2. The "Day Zero" Test:

   • Check: If a developer followed ONLY these instructions, what crucial configuration or setup step is missing?
   • Suggestion Pattern: "Improve implementation readiness by including mandatory [Dependency/Config] setup steps."

3. The "Constraint Hardening" Check:

   • Check: Does the skill clearly state what NOT to do (Hard Constraints)?
   • Suggestion Pattern: "Reduce risk by explicitly documenting negative constraints (e.g., 'DO NOT use [Forbidden Library]')."

4. The "Verification Loop":

   • Check: Does the skill provide a way for the user (or another agent) to verify the result is correct?
   • Suggestion Pattern: "Enhance reliability by adding a 'Verification' section with specific test cases or shell commands."

Every audit report MUST include at least one actionable suggestion from each pillar where a gap is identified. This ensures consistent, high-fidelity coaching across different sessions.

Include best practices and known issues that may be over looked.