Ss AI Audit
Testing & Qualityss-ai-auditSs AI Audit is an Agent Skill for testing & quality published in the Serial-Studio/Serial-Studio repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Audit Serial Studio's AI-facing material (CLAUDE.md, doc/claude/**, .claude/skills/**, and optionally the in-app assistant corpus under app/rcc/) against code ground truth. Use when asked to "audit the skills/docs", "check the docs for drift", "verify CLAUDE.md is still true", or after a refactor that moved/renamed things (TU splits, workflow consolidation, symbol renames). Finds stale claims and fixes them with targeted edits; docs-only, never touches code.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/Serial-Studio/Serial-Studio/blob/HEAD/.claude/skills/ss-ai-audit/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/ss-ai-audit, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Serial Studio — AI-facing doc audit
AI-facing docs steer future edits only while they are true; a stale claim is worse than no claim because it gets verbalized with confidence at the point of action. Code evolves, docs don't — this skill is the periodic re-sync. Two sweeps have been needed already (2026-06 corpus sweep, 2026-07 skills audit), so treat drift as expected, not exceptional.
Procedure
-
Enumerate the claim-bearing files in scope:
CLAUDE.md,doc/claude/**(includingarchitecture/),.claude/skills/*/SKILL.md+references/, and — if asked — the in-app assistant material underapp/rcc/. -
Extract the verifiable claims. A claim is anything checkable against the repo: a file path, a symbol (
Class::method), a numeric constant, a CLI flag or default, a workflow file name, a pipeline step list, an enum value list, a registration touch-point list, a "this file contains X" example. Skip opinions and rules — audit facts. -
Verify each claim against ground truth with
Grep/Read— the code is the truth, the doc is the suspect. For volume, fan out parallel read-only subagents, one per skill or subsystem, each with an explicit numbered claim list and a VERIFIED / WRONG (+ correct fact) / NOT FOUND verdict format withfile:lineevidence. Keep the missions disjoint (doc/claude/j-space.md, named lenses). -
Fix with targeted edits — and fix every mirror. Wrong facts propagate: the same claim typically lives in CLAUDE.md, a
doc/claude/sub-doc, and one or more skills. After correcting one instance, grep the wrong literal across all AI-facing files and fix every copy in the same pass. Before each fix, state the ground-truth evidence (file:line) in chat — a correction you can't evidence is a new guess. -
Report what was verified clean vs corrected, with the evidence. Do not silently fix.
Known drift magnets (check these first)
- Moved implementations — TU splits and renames leave docs pointing at the old file
(e.g.
ProjectEditor.cpp→Project/ProjectEditorShared.h+ProjectEditorForms.cpp). - CI workflow names — jobs get consolidated/renamed (
test.yml/deploy.yml→ci.yml). - Interface vs concrete attribution — a method documented on the interface that only the
concrete class carries (
IScriptEngine::guardedCall→JsScriptEngine::guardedCall). - Look-alike constants — two real numbers conflated (65536 enqueue queue vs 4096
kCapturedPoolSize). Verify which structure the number belongs to, not just that it appears in code. - Pipeline/checklist step lists — scripts grow steps (
black,generate-sdk) that the doc's enumeration silently omits. - "Known bug" examples — landmines get fixed; a doc citing one as present is stale
(reframe as "was fixed; flag reintroduction"), and "e.g. in file X" sightings get cleaned
up (
Q_FOREACHinBluetoothLE.cpp). - Registration touch-point lists — new switches/resources appear (icon
.qrcentries, extraBusTypeswitches); re-grep a recently added enum value and mirror every hit.
Rules
- Docs-only. Never "fix" code to match a doc — if the code looks wrong, say so in chat; the doc records reality, the maintainer changes reality.
- Every correction carries
file:lineground-truth evidence. - Stay in the AI-facing lane; user-facing docs (
README.md,doc/help/**) have their own linter (documentation-verify.py) and tone rules — don't restyle them from here. - When a skill's facts change, check
doc/claude/repo-skills.mdand CLAUDE.md's pointers to it still describe it correctly.