validate-sarif

Validate SARIF Findings

Context

SARIF files produced by AI security agents must conform to three layers of correctness:

1. Base schema layer — Valid SARIF 2.1.0 per the OASIS standard. Structural issues like misplaced properties, missing required fields, or type mismatches. Sarif.Multitool validate checks this.

2. Whole-log overlay layer — ai-sarif-log.schema.json, the post-enrichment output schema Sarif.Multitool get-schema serves for emit-finalize. It $refs base SARIF 2.1.0 and tightens it to the subset of Error-level AI rules JSON Schema can express at whole-log scale: the result.ruleId shape (CWE-<n>/<sub-id>, bare CWE-<n>, or NOVEL-<sub-id>), message.markdown non-empty, a physicalLocation with region.startLine >= 1, a per-run versionControlProvenance (so a --no-repo log is rejected — it is unpublishable), properties[ai/origin] in the closed set, and the GHAzDO automationDetails contract when the Azure DevOps pipeline shape is present. This overlay is advisory — the contract get-schema documents, not a gate validate runs.

3. AI rich-rule layer — The conventions in docs/ai/generating-sarif.md, enforced by the multitool's AI rule pack (AI1003–AI2019) under --rule-kind "Sarif;AI". These carry every semantic rule JSON Schema cannot express; this skill rounds out the remaining profile-level checks.

All three matter, and the boundary between layers 2 and 3 is deliberate. JSON Schema validates structure and shape; it cannot reach the semantics below. The overlay accepts these, and only the rich rules (validate) reject them:

• Closed vocabularies on the open properties bag — ai/exploitability: "unconfirmed" is structurally a string, so the overlay passes it; AI2014 holds it to the {demonstrated, poc, theoretical} set.
• All-or-nothing co-presence — a lone ai/exploitability with no ai/attackerPosition/ai/evidence companion passes the overlay; AI2014 requires the triple to be all-present or all-absent.
• Cross-run grouping reciprocity — the overlay validates each run independently, so a one-directional grouping link passes; AI1015 requires it to be reciprocal.
• Non-persistence — a rolling-hash partialFingerprints entry is a well-formed string map to the overlay; AI2011 forbids persisting it as a stable identity.
• Category vs. Weakness ruleId base — CWE-16/insecure-default-config matches the overlay's structural CWE-<n>/<sub-id> shape, but CWE-16 is a MITRE Category, never a valid mapping target. Whether a base CWE is a Category or a Weakness is taxonomy data the schema does not carry; the Category-mapping rule rejects it as a hard producer-emit failure. A Category ruleId is an emit bug, not something to normalize — never auto-correct it to a Weakness or a name floor.

A structurally valid SARIF file with ai/exploitability: "unconfirmed" passes both schema layers but fails downstream consumers; conversely, a file with correct ai/* keys but contextRegion nested inside region (instead of physicalLocation) is structurally broken at layer 1.

The CLR seam — which layers you can enforce depends on your runtime. Layers 1 and 2 are portable JSON Schema: any consumer with a JSON Schema (Draft 2020-12) validator can enforce them with no .NET dependency, including the TypeScript-only @microsoft/sarif library. ai-sarif-log.schema.json is therefore the ceiling of validation a no-CLR consumer gets for free — it captures every AI rule JSON Schema can express. Layer 3, the rich-rule pack, requires the CLR: take a dependency on Sarif.Multitool and run validate --rule-kind "Sarif;AI". If you can take the CLR, do — it is the only way to get the semantic rules above (reciprocity, co-presence, closed vocabularies, Category-vs-Weakness, non-persistence). A TS-only consumer that cannot take the CLR validates the full schema (base SARIF 2.1.0 + ai-sarif-log.schema.json) and either accepts the layer-3 gaps or, if it wants that coverage, implements those semantic checks itself. One layer-3 gap it can close directly: @microsoft/sarif-multitool-ts exports isCweCategory(id) / getCweCategoryName(id) / getCweCategories() (backed by the bundled CweCategories.json), so a TS consumer can flag a Category-mapped ruleId without the CLR. Categories do not appear in CweTaxonomy.sarif / getCweTaxonomy() and cannot be inferred from cwe/abstraction — use that API, not the taxonomy.

Why validate early: every SARIF file that reaches a result store, dashboard, or remediation agent with profile violations creates silent data quality debt. Validate at production time — immediately after the producing agent writes the file — to catch issues before they propagate.

Prerequisites

• Sarif.Multitool ≥ 5.0.0. Recommended invocation: dotnet dnx Sarif.Multitool --yes -- validate ... (zero-install, requires .NET 10+). Fall back to dotnet tool install --global Sarif.Multitool if dotnet dnx is unavailable.
• The normative profile document at docs/ai/generating-sarif.md. This skill references it but embeds the rules for offline execution.
• The standard (base-profile) validation rule catalog at docs/ValidationRules.md. AI- and GHAzDO-profile rules are documented in docs/ai/generating-sarif.md.

Detection

Inputs

If both {{SARIF_FILE}} and {{SARIF_DIR}} are provided, validate the single file. If neither is provided, ask the operator.

Step 1 — Schema + SDK AI Validation (Sarif.Multitool)

Run the multitool with --rule-kind "Sarif;AI" to activate both standard SARIF rules and the SDK's AI profile rule pack in one pass:

dotnet dnx Sarif.Multitool --yes -- validate "{{SARIF_FILE}}" --rule-kind "Sarif;AI" --level "Note;Warning;Error"

CRITICAL: Always use --rule-kind "Sarif;AI". Without AI in the list, the AI rule pack (AI1003–AI2019) is opt-in and won't run. Without Sarif, you skip the standard rules (SARIF1xxx / SARIF2xxx / JSON1xxx) that catch structural drift the AI rules don't cover.

Capture all output. Each line with note, warning, or error is a finding. The following AI rules are implemented:

General SARIF rules that commonly fire on AI-generated output:

See docs/ValidationRules.md for the standard catalog; AI- and GHAzDO-profile rules are documented in docs/ai/generating-sarif.md.

If dotnet dnx is not available: Fall back to the global tool: dotnet tool install --global Sarif.Multitool then run:

sarif validate "{{SARIF_FILE}}" --rule-kind "Sarif;AI" --level "Note;Warning;Error"

If neither dotnet approach works, report:

Sarif.Multitool is not available. Install .NET 10+ for dotnet dnx, or run: dotnet tool install --global Sarif.Multitool

and skip to Step 2 (AI profile checks can still run independently). Note: Step 2 MUST include a startLine >= 1 check (rule JSON1008 equivalent) since schema validation was skipped — a startLine: 0 that passes AI checks but fails schema validation is a silent defect.

Step 2 — AI Profile Validation

Parse the SARIF JSON and check each rule below. These correspond to the rules in docs/ai/generating-sarif.md § Appendix: Validation Rules.

If {{PROFILE}} is schema-only, skip this step entirely.

Run-level checks

Result-level checks (for each result)

All-or-nothing consistency checks

Evidence checks

Key count check

Notification checks

Step 3 — Report

Produce a structured report. Group findings by severity:

## SARIF Validation Report

**File:** {{SARIF_FILE}}
**Schema validation:** ✅ PASS | ❌ N errors, M warnings
**AI profile validation:** ✅ PASS | ❌ N errors, M warnings, K notes

### Errors (must fix)
- AI2014: result[0] — ai/exploitability value "unconfirmed" is not in {demonstrated, poc, theoretical}
- ...

### Warnings (should fix)
- ...

### Notes (consider)
- ...

### Summary
| Layer | Errors | Warnings | Notes |
|---|---|---|---|
| Schema (Multitool) | 0 | 0 | — |
| AI Profile | 0 | 0 | 0 |
| **Total** | **0** | **0** | **0** |

Edge Cases

1. Multiple runs in one log — Validate each run independently. Report run index in each finding.
2. Redacted vs full log pairs — If ai/redacted: true, expect fewer keys (no ai/handoff, ai/evidence, tool-namespace keys in redacted copy). Validate the redacted log against the redacted subset of rules. If ai/fullLogLocation is present, note it but do not attempt to fetch the full log.
3. Empty results array — Valid SARIF. All-or-nothing rules are vacuously satisfied. Note "0 results — nothing to profile-check."
4. Non-AI SARIF — If ai/origin is absent from all runs, report AI1006 as an error but skip result-level AI checks (the file is not claiming to be AI-produced).
5. Very large files — Parse with streaming if >50MB. The multitool handles this; for profile checks, read results incrementally if memory is a concern.

Known Drift Patterns

AI agents generating SARIF systematically drift from the standard in predictable ways. This catalog captures observed patterns — use it both for validation (catching drift) and for improving the emit-sarif skill (preventing drift).

How drift happens: LLMs generate SARIF from training data that includes pre-standard drafts, partial examples, and SARIF from non-AI tools. The emit-sarif skill instructs them correctly, but agents hallucinate "reasonable" values that aren't in the vocabulary, or place properties at plausible-but-wrong locations in the object graph. Schema validation catches structural drift (#1, #6); AI profile rules catch semantic drift (#2, #4, #5, #8, #10). Both layers are needed.

Feedback loop: When this skill finds a new drift pattern not in this catalog, add it. When the emit-sarif skill is updated to prevent a pattern, note that in the catalog but keep the validation rule — prevention at generation time reduces but never eliminates drift.

Validation

After running this skill:

1. Every error-level finding should be actionable — the producing agent can fix it.
2. The report should be self-contained (no need to cross-reference the normative doc).
3. Zero false positives on well-formed files — validate against docs/ai/example.sarif as a smoke test.

Escalation

• Multitool not available — Neither dotnet dnx nor global tool install succeeded. Report the gap, run AI profile checks only (Step 2), note that schema validation was skipped.
• SARIF file is not valid JSON — Report parse error and stop; no further validation is possible.
• Unknown ai/* keys found — Report them by name; they may be legitimate tool-specific extensions using an ai/ prefix incorrectly, or they may indicate a profile evolution this skill doesn't know about yet. Recommend checking whether docs/ai/generating-sarif.md has been updated.