pre-deploy-qa

Pre-deploy QA

Input Requirements

Read before starting:

• user-spec.md — acceptance criteria
• tech-spec.md — technical acceptance criteria
• decisions.md — deviations from plan (if exists)
• Project Knowledge — architecture.md, patterns.md (incl. Testing & Git Workflow sections)

If user-spec or tech-spec missing — request before proceeding.

Verification Directions

Three verification directions (order doesn't matter):

Test Suite

Run all tests (unit, integration, E2E). All must pass.

• Identify test runner from project config (package.json, pyproject.toml, Makefile, etc.)
• Run full test suite
• Record: total tests, passed, failed, skipped

Acceptance Criteria

Check every criterion from user-spec and tech-spec:

• passed — criterion met, evidence provided
• failed — the feature exists but does not meet the criterion
• not_verifiable — cannot be checked without live environment, external service, or MCP tool (scope of post-deploy-qa)

For each criterion — provide evidence (test name, code path, log output).

Coverage Verification

After test suite passes, verify that tests actually exercise the feature:

• For each file in the feature's scope (from tech-spec "Files to modify"): verify a corresponding test exists. Feature code without any test → severity critical
• If project has coverage tooling configured (jest --coverage, pytest --cov, vitest --coverage) — run it. Coverage of feature files dropping below project threshold → severity critical
• For each acceptance criterion with status passed — verify the linked test actually exercises the relevant code path, not just an import check or mock-only test. Test that doesn't actually test the feature behavior → severity major
• Edge cases mentioned in user-spec (error handling, boundary values, empty states) — verify they have corresponding tests. Missing edge case test for M/L features → severity major

Severity Classification

• critical — acceptance criterion failed, tests fail, core functionality broken
• major — works but with significant issues (edge cases, UX bugs, degraded behavior). Escalate to critical if it affects data integrity or core user workflow.
• minor — cosmetic, inaccuracies, improvements

Output

JSON report → logs/working/qa-report.json

Full report saved to file. Reason: orchestrator parses this to decide pass/fail.

{
  "status": "passed | failed",
  "summary": {
    "totalChecks": 0,
    "passed": 0,
    "failed": 0,
    "notVerifiable": 0,
    "criticals": 0,
    "majors": 0,
    "minors": 0
  },
  "testSuite": {
    "status": "passed | failed",
    "details": "All 42 tests passed"
  },
  "acceptanceCriteria": [
    {
      "criterion": "User can login with email",
      "status": "passed | failed | not_verifiable",
      "evidence": "Test login_test.py::test_email_login passes"
    }
  ],
  "findings": [
    {
      "severity": "critical | major | minor",
      "title": "Login fails for emails with + sign",
      "expected": "Login succeeds",
      "actual": "400 Bad Request",
      "reproduction": "Steps to reproduce..."
    }
  ]
}

Status decision: passed if zero criticals, failed if one or more criticals.

decisions.md entry — concise summary only

Write a brief entry to decisions.md following the template (~/.claude/shared/work-templates/decisions.md.template). Link to logs/working/qa-report.json for the full report.

Example:

## Task 9: Pre-deploy QA

**Status:** Done
**Agent:** qa-runner
**Summary:** QA passed. 391 tests green, 28 acceptance criteria checked (25 passed, 3 not_verifiable). No blockers.
**Deviations:** None.

**Verification:**
- Full report: [logs/working/qa-report.json]

Guidelines

• Work from specs only (user-spec, tech-spec, decisions.md). Task files (tasks/*.md) are already verified by reviewers and are outside QA scope.
• Account for decisions.md — deviations from original plan may be justified.
• Every finding includes concrete reproduction: steps, expected vs actual.
• Criteria requiring live environment or MCP tools — mark as not_verifiable, note that post-deploy verification is needed.
• Empty findings array = clean audit.

Deferred to Post-deploy

If any acceptance criteria are marked not_verifiable — add a deferredToPostDeploy section to the JSON report. This section is the handoff contract: post-deploy QA reads it and verifies each deferred criterion on live environment.

For each deferred criterion, specify:

• Which criterion (ID and text)
• Why it cannot be verified pre-deploy
• What conditions are needed to verify it (live data, MCP tool, user action)
• Concrete verification steps for post-deploy agent

Example in JSON report:

"deferredToPostDeploy": [
  {
    "criterion": "US-5: Titles generated with correct declensions",
    "reason": "Requires live LLM call with real data",
    "verificationCondition": "New survey entry processed after deploy",
    "verificationSteps": "Run a survey entry through the bot, check generated title for grammar and naturalness"
  }
]

Also mention deferred criteria in the decisions.md entry:

**Deferred to post-deploy:** 3 criteria require live verification (US-5, US-8, US-10). See deferredToPostDeploy in qa-report.json.