cubic-audit

cubic-audit

Periodic audit of the cubic AI-review configuration (cubic.yaml at the repo root). Two deliverables, in order:

1. Rule review: are the custom review rules the highest-signal set for this repo, correctly scoped, within cubic's limits, and factually accurate against the code?
2. Learnings reconciliation: which accumulated cubic learnings should be promoted into durable rules, and which are obsolete / duplicate / contradictory and should be cleaned up in the cubic UI.

Two modes. Deliverable 2 needs the cubic learnings MCP (tools list_learnings / get_learning). That server is connected per-account in the cubic app, not declared in the repo's .mcp.json, so a teammate running this skill may not have it. Probe for it in step 1: if it is reachable, run the full audit (both deliverables). If not, run rules-only mode (deliverable 1 plus the cubic.yaml validation in step 4), and tell the user the learnings reconciliation was skipped because the cubic MCP is not connected, with a one-line pointer on how to enable it (connect the cubic integration in the cubic app, then re-run). Never block the whole skill on the missing MCP.

Cubic docs (re-check if a limit below seems wrong): https://docs.cubic.dev/configure/cubic-yaml and https://docs.cubic.dev/ai-review/custom-agents

Language: all output (chat, cubic.yaml text, commit messages) is US English, regardless of the language used to invoke the skill.

Open source: cartography is a public CNCF project. Never reference clients, internal tickets, or private deployments in cubic.yaml, commits, or learnings recommendations.

Hard constraints (bake these in)

• Max 5 enabled custom rules per repo. Order matters; beyond the limit only the first N take effect. To add one you must merge or drop one. cubic.yaml is already at the cap (5 custom_rules), so every "promote" recommendation must name what it merges into or replaces.
• 10,000 characters per rule, and that budget includes the resolved content of any file_paths. Attaching a large doc (AGENTS.md, a schema.md) is counterproductive: it truncates to the least-relevant top and starves the prompt. Prefer self-contained prompts; only link a file if it is small (<2k) and review-focused.
• cubic.yaml merges across levels: repo > org > UI > built-in defaults.
• No MCP tool deletes or edits a learning. Learnings cleanup is a manual action in the cubic UI. This skill produces the list; a human actions it.
• No em-dashes / en-dashes anywhere in cubic.yaml or commits. Run grep -n '—\|–' cubic.yaml before declaring done.
• Honor the repo's worktree/path and git rules in AGENTS.md (CLAUDE.md may be a symlink to it; read AGENTS.md directly). Commit/push only when explicitly asked; commits use --signoff (DCO is required on this repo); never open a PR without per-PR authorization.

Workflow

0. Context

• Confirm the repo root (or worktree). Read cubic.yaml.
• Derive the GitHub slug: git remote get-url origin -> owner/repo (this repo: cartography-cncf/cartography).

1. List learnings (cubic MCP) -- skip if unavailable

The cubic MCP server is namespaced by an opaque id and is connected per-account in the cubic app, not in the repo's .mcp.json. Probe first: run ToolSearch for list_learnings get_learning cubic. If it returns no matching tool, the cubic MCP is not connected in this environment: skip the rest of this step and all of steps 2 and 5's cleanup table, run rules-only mode (steps 3-4), and state in the report that learnings reconciliation was skipped (connect the cubic integration in the cubic app and re-run to enable it). Do not fabricate learnings from memory.

If the tool is found, call list_learnings(owner, repo).

The result is large and is persisted to a file (the tool message returns its path). Do not read it raw. Extract a compact view and read 100% of it:

# f = the persisted list_learnings result file path from the tool message
awk '
/^## / { cat=$0; sub(/^## /,"",cat); next }
/^### / { title=$0; sub(/^### /,"",title); desc=""; getline; while($0=="") getline; while($0!="" && $0 !~ /^- \*\*/){ desc=desc $0 " "; getline } next }
/^- \*\*Confidence\*\*/ { c=$0; sub(/.*: /,"",c) }
/^- \*\*Source\*\*/     { s=$0; sub(/.*: /,"",s) }
/^- \*\*Updated\*\*/    { u=$0; sub(/.*: /,"",u); printf "[%s] %s | %s | %s | %s\n    %s\n", cat, title, c, s, u, desc }
' "$f" > /tmp/cubic_learnings_compact.txt
wc -l /tmp/cubic_learnings_compact.txt

Read the compact file fully (chunk it). Never pipe the awk through head: it silently drops the tail and you will miss whole categories.

2. Bucket the learnings

Most learnings are negative / suppression ("Do not flag X", "Treat Y as intentional", "Before flagging Z, check W"). These are false-positive corrections that already apply automatically. Do NOT turn suppression learnings into rules: a rule says "flag", a suppression says "do not".

• Promote candidates: positive / prescriptive ("always do X"), general, recurring (appears 2+ times or echoes a known convention), and not already covered by an existing rule. Expect only 1-3 per audit. Since the config is at the 5-rule cap, each promotion must fold into an existing rule's description rather than add a sixth.
• Cleanup candidates:
  • Contradictory: conflicts with a rule or with repo reality. Highest priority, since these suppress real findings.
  • Obsolete: references a removed/renamed module, deleted helper, or a label/pattern that no longer exists in the code.
  • Duplicate: near-identical pairs (often the same rule reworded, or one with a hyphen vs an arrow).
  • Low-value: vague entries under ~75% confidence that are platitudes. Optional prune.

Verify before recommending deletion. Never trust the learning text alone: grep the code for the referenced symbol / dir / feature. Patterns that pay off:

git grep -n 'def from_node_schema'             # the signature a rule quotes is correct?
git grep -lI 'aws_handle_regions' -- '*.py'    # decorator still referenced?
ls cartography/intel/<module> 2>/dev/null      # module still exists / not renamed?

3. Audit the rules themselves

• Are the (<=5) rules the highest-signal for this graph-centric (Neo4j/Cypher) intel tool? Rules tied to the declarative data model (node/rel schemas, MatchLinks, scoped cleanup) and the get -> transform -> load -> cleanup sync contract earn their slots; pure-hygiene rules are the first to cut when you need room.
• Redundant with tooling? Drop checks already enforced by the pre-commit stack (black, isort, flake8, pyupgrade, mypy). Example: pyupgrade --py36-plus already rewrites legacy typing.Dict/List/Optional to PEP 585 builtins, so a rule should not also police that. Inspect .pre-commit-config.yaml and the [flake8] block in setup.cfg before keeping a hygiene rule.
• Scoping: every rule should carry include/exclude globs so it only runs where relevant (cuts false positives, saves the 10k budget). Test-only guidance -> tests/**; schema-doc guidance -> docs/root/modules/**; AWS-decorator guidance -> cartography/intel/aws/**. The existing PII rule already excludes tests/** as a model.
• Factual accuracy: every helper name / signature / path quoted in a prompt must match the code. Verify with grep (a wrong signature in a prompt produces wrong reviews). Spot-check the claims already baked into cubic.yaml (e.g. GraphJob.from_node_schema(), GraphJob.from_matchlink(), @timeit, @aws_handle_regions, the RESOURCE/INWARD sub-resource contract).
• ignore.files (use sparingly): reviews.ignore.files skips matching files entirely, so it removes all review coverage, not just noise. Reserve it for purely machine-generated artifacts with no hand-authored content. Do not ignore files that another mechanism already governs or that carry meaningful diffs: in this repo uv.lock must stay reviewable because a custom rule flags unwarranted lockfile-only churn (ignoring it disables that signal), and tests/data/** are hand-maintained fixtures that AGENTS.md lists as part of the intel-module surface. Prefer a scoped custom rule over hiding a file. The current cubic.yaml intentionally has no ignore block.

4. Validate cubic.yaml

uv run --with pyyaml --no-project python -c "
import yaml
d=yaml.safe_load(open('cubic.yaml')); r=d['reviews']
print('YAML OK | rules:', len(r['custom_rules']), '(max 5)')
for c in r['custom_rules']:
    n=len(c['description']); print(f\"  {n:>5} chars  {c['name']}{'  <-- OVER 10k!' if n>10000 else ''}\")
"
grep -n '—\|–' cubic.yaml && echo 'FIX em/en-dashes' || echo 'no em/en-dashes'

5. Report, then apply

Present three recap tables and stop for approval before editing:

• Promote: learning -> target rule (which existing rule it folds into) -> why.
• Cleanup (grouped: contradictory / obsolete / duplicate / low-value): learning -> type -> code-verified justification.
• Refactor: rule changes (merge / split / rescope) within the 5 cap.

After approval:

• Apply the cubic.yaml edits, then re-run the step-4 validation.
• Hand the learnings cleanup list to the user as manual cubic-UI actions (no MCP delete exists).
• Commit/push only if asked; PR only with per-PR authorization. Commit message in US English, with --signoff, plus the repo's standard Co-Authored-By trailer.

Notes

• Read-mostly on cubic: this skill lists learnings, never mutates them, and edits only cubic.yaml.
• Keep rule prompts self-contained and concrete (good/bad examples + the "why"); that beats terse rules and avoids the file_paths truncation trap.