ct-grade

Session Grading Guide

Session grading evaluates agent behavioral patterns against the CLEO protocol. It reads the audit log for a completed session and applies a 5-dimension rubric to produce a score (0-100), letter grade (A-F), and diagnostic flags.

When to Use Grade Mode

Use grading when you need to:

• Evaluate how well an agent followed CLEO protocol during a session
• Identify behavioral anti-patterns (skipped discovery, missing session.end, etc.)
• Track improvement over time across multiple sessions
• Validate that orchestrated subagents followed protocol

Grading requires audit data. Sessions must be started with the --grade flag to enable audit log capture.

Starting a Grade Session

CLI

# Start a session with grading enabled
ct session start --scope epic:T001 --name "Feature work" --grade

# The --grade flag enables detailed audit logging
# All CLI operations are recorded for later analysis

Running Scenarios

The grading rubric evaluates 5 behavioral scenarios that map to protocol compliance:

1. Fresh Discovery

Tests whether the agent checks existing sessions and tasks before starting work. Evaluates session.list and tasks.find calls at session start.

2. Task Hygiene

Tests whether task creation follows protocol: descriptions provided, parent existence verified before subtask creation, no duplicate tasks.

3. Error Recovery

Tests whether the agent handles errors correctly: follows up E_NOT_FOUND with recovery lookups (tasks.find), avoids duplicate creates after failures.

4. Full Lifecycle

Tests session discipline end-to-end: session listed before task ops, session properly ended, CLI usage patterns.

5. Multi-Domain Analysis

Tests progressive disclosure: use of admin.help or skill lookups, use of progressive disclosure for programmatic access.

Evaluating Results

CLI

# Grade a specific session
ct grade <sessionId>

# List all past grade results
ct grade --list

Understanding the 5 Dimensions

Each dimension scores 0-20 points, totaling 0-100.

S1: Session Discipline (20 pts)

What it measures: Does the agent check existing sessions before starting, and properly close sessions when done?

S2: Discovery Efficiency (20 pts)

What it measures: Does the agent prefer tasks.find (low context cost) over tasks.list (high context cost) for discovery?

S3: Task Hygiene (20 pts)

Starts at 20 and deducts for violations:

What it measures: Does the agent create well-formed tasks with descriptions and verify parents before creating subtasks?

S4: Error Protocol (20 pts)

Starts at 20 and deducts for violations:

What it measures: Does the agent recover gracefully from errors and avoid creating duplicate tasks?

S5: Progressive Disclosure Use (20 pts)

What it measures: Does the agent use progressive disclosure (help/skills) for efficient protocol access?

Interpreting Scores

Letter Grades

Reading the Output

The grade result includes:

• score/maxScore: Raw numeric score (e.g., 85/100)
• percent: Percentage score
• grade: Letter grade (A-F)
• dimensions: Per-dimension breakdown with score, max, and evidence
• flags: Specific violations or improvement suggestions
• entryCount: Number of audit entries analyzed

Flags

Flags are actionable diagnostic messages. Each flag identifies a specific behavioral issue:

• session.list never called -- Check existing sessions before starting new ones
• session.end never called -- Always end sessions when done
• tasks.list used Nx -- Prefer tasks.find for discovery
• tasks.add without description -- Always provide task descriptions
• Subtasks created without parent existence check -- Verify parent exists first
• E_NOT_FOUND not followed by recovery lookup -- Follow errors with tasks.find
• No admin.help or skill lookup calls -- Load ct-cleo for protocol guidance
• No progressive disclosure calls -- Use admin.help or skill lookups

Common Anti-patterns

Grade Result Schema

Grade results are stored in .cleo/metrics/GRADES.jsonl as append-only JSONL. Each entry conforms to schemas/grade.schema.json with these fields:

• sessionId (string, required) -- Session that was graded
• taskId (string, optional) -- Associated task ID
• totalScore (number, 0-100) -- Aggregate score
• maxScore (number, default 100) -- Maximum possible score
• dimensions (object) -- Per-dimension { score, max, evidence[] }
• flags (string[]) -- Specific violations or suggestions
• timestamp (ISO 8601) -- When the grade was computed
• entryCount (number) -- Audit entries analyzed
• evaluator (auto | manual) -- How the grade was computed

CLI Grade Operations