agent-eval-flywheel

Agent Eval Flywheel Skill

How Google grades AI agents, and how you drive the loop. You pick a metric that matches what you care about, run the eval, read what the judge says, and refine. The payoff is a judge that scores an answer's meaning and explains its verdict in plain English, a score you can act on.

[!IMPORTANT] Two skills ship this workflow. google-agents-cli-eval drives ADK agents through the agents-cli toolchain (in agents-cli 0.1.1 eval run shells out to adk eval, in 1.0.0 it chains eval generate and eval grade). agent-platform-eval-flywheel is the framework agnostic sibling for the cloud Evaluation SDK, published in google/skills under skills/cloud/. This skill covers the local ADK path, which needs nothing but the Gemini API key.

[!IMPORTANT] The metric menu is the whole mental model. Two metrics run LOCALLY and free, tool_trajectory_avg_score (did it call the right tools) and response_match_score (does the text match a reference). The rest are LLM as judge, final_response_match_v2, rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1, hallucinations_v1, and safety_v1. First decision every time, do you care about the exact words or the meaning. Words, use a local metric. Meaning, use a judge.

---

Quick Start

The full eval fix loop, all verified against google-adk 2.4.0 on Python 3.12 with the Gemini API key.

# 1. Environment. The agent AND the judge both call the Gemini API.
export GOOGLE_GENAI_USE_VERTEXAI=0
export GOOGLE_API_KEY=YOUR_AI_STUDIO_KEY

# 2. Grade with the free local metrics.
adk eval ./episode_finder tests/eval/evalsets/episode_finder.evalset.json \
  --config_file_path tests/eval/eval_config.json --print_detailed_results
#   tool_trajectory_avg_score  PASSED 1.0   <- it called the right tool

# 3. Add an AI judge. It scores MEANING, not words.
adk eval ./episode_finder tests/eval/evalsets/episode_finder.evalset.json \
  --config_file_path tests/eval/eval_config_judge.json --print_detailed_results
#   final_response_match_v2    PASSED 1.0

# 4. Close the loop. Keep tool trajectory plus the judge, rerun, all green.
adk eval ./episode_finder tests/eval/evalsets/episode_finder.evalset.json \
  --config_file_path tests/eval/eval_config_fixed.json --print_detailed_results

[!NOTE] response_match_score is ROUGE, it compares words. A correct answer worded differently from your reference scores low (measured 0.2 to 0.35 on this agent, always under the 0.5 threshold). That is not a broken agent, it is the signal to use a judge when you care about meaning, not exact wording.

[!WARNING] adk eval needs the eval extra. A bare pip install google-adk gives Error: Eval module is not installed, please install via pip install "google-adk[eval]". Install google-adk[eval].

[!WARNING] The system macOS Python is 3.9, google-adk needs 3.10 or newer. Stand up a current interpreter, uv venv --python 3.12, before installing.

The judge that explains itself

The best part of the toolkit. A rubric judge grades against plain-English rules you write, and writes a reason for each one, against the trusted answer. adk eval prints scores but drops the rationale into a history JSON under AGENT_DIR/.adk/eval_history/, so eval_report.py surfaces it.

python eval_report.py ./episode_finder \
  tests/eval/evalsets/episode_finder.evalset.json \
  --config tests/eval/eval_config_rubric.json

Real output, the judge grading against the trusted evidence.

Q  Which Stage Studio episode covered background agents?
A  Episode e02, titled "Managed Agents background and remote MCP", covered background agents.
   rubric_based_final_response_quality_v1: PASSED score=1.0
      why [names_episode_id] 1.0: states "Episode e02", consistent with the trusted evidence.
      why [names_title] 1.0: includes the title "Managed Agents background and remote MCP".
      why [no_guessing] 1.0: only the episode id and title from the find_episode tool, nothing invented.

[!IMPORTANT] The judge is decoupled by design, Google's rule. Nothing grades its own work, the agent does not score its answers and the optimizer does not score its fixes. A separate judge model call does the grading against the reference and writes the reason. Keeping those two roles apart is the point of the flywheel.

Optimize and iterate, catching a hallucination

This is the payoff, and the failure it catches is the one everyone worries about, the agent making something up. The judge tells you WHY a case failed, you apply one targeted fix, and rerun to prove it helped. Google's own framing, it proposes, you approve, human in the loop, not hands off. The fix is almost always an edit to the agent's instruction, not the model.

Start from a well meaning but ungrounded agent (optimize/finder). Its instruction pushes it to always sound confident and never leave a viewer empty handed. Ask it about topics the show has not covered (optimize/grounding.evalset.json), and grade it with a grounding rubric (optimize/eval_config_grounding.json). It invents episodes that do not exist, and the judge catches every one.

BEFORE, ungrounded agent, asked about a topic the tool did not find
Q  Which episode covered quantum computing?
A  We covered quantum computing in Episode 77: "Quantum Leaps and Bounds: The Future of Computing."
   rubric_based_final_response_quality_v1: FAILED score=0.0
     why [grounded] 0.0: the tool returned found=false, but the answer invented "Episode 77", contradicting the evidence.

Now the proposing part. propose_fix.py reads the judge's failing reason and the agent's current instruction, and asks a model to propose a revised instruction. The proposer is deliberately NOT the judge, same decoupling rule.

python propose_fix.py ./optimize/finder optimize/grounding.evalset.json \
  --config optimize/eval_config_grounding.json

=== Proposed instruction (a model wrote this, NOT the judge) ===
You are the Stage Studio episode helper. When a viewer asks which episode covered a topic,
answer with the episode number and title. If the topic has not been covered, clearly state
that it has not and do not invent an episode.

Approve it, paste it into the agent's instruction (the grounded version ships in finder_fixed/), and rerun the same eval. The loop closes.

AFTER, grounded agent, same question
Q  Which episode covered quantum computing?
A  We have not covered quantum computing yet.
   rubric_based_final_response_quality_v1: PASSED score=1.0
     why [grounded] 1.0: says the topic has not been covered, invents no episode. Grounded.

That is one turn of the flywheel, read the reason, apply a targeted fix, rerun, compare to the baseline. Repeat until the case passes, expect several turns on a real agent.

[!NOTE] An automated version of this turn exists. google-adk 2.4.0 ships adk optimize, which refines the root agent instructions with the GEPA framework against a target metric, and agents-cli 1.0.0 exposes it as the experimental eval optimize. Both are confirmed from the tools' own help output, not yet exercised by this skill's verified runs, treat them as experimental.

Workflow

When asked to evaluate an ADK agent, follow this loop.

1. Analyze the goal. Care about MEANING, reach for an LLM judge. Care about the exact tool sequence, use tool_trajectory_avg_score, local and free.
2. Run eval_report.py AGENT_DIR EVALSET --config CONFIG so you see the judge's reasons, not just red or green numbers.
3. Read the reason, make one change, rerun. Expect several iterations on a real agent, that is normal, each pass makes it better.

Dependencies and Prerequisites

• Python 3.10 or newer (uv venv --python 3.12 if the system Python is 3.9).
• google-adk[eval] >= 2.4.0, install uv pip install "google-adk[eval]".
• agents-cli optional, uv tool install google-agents-cli. Verified at 0.1.1 where eval run shells out to adk eval. The 1.0.0 release restructures the group, eval run chains eval generate and eval grade, and adds experimental eval analyze (loss clusters, needs a GCP project) and eval optimize (runs adk optimize, GEPA prompt refinement).
• A Gemini API key for both the agent and the judge, or Vertex via GOOGLE_GENAI_USE_VERTEXAI=1 and a project.

[!NOTE] Automatic Loss Analysis (failure clustering) and adaptive AutoRaters that write a feedback paragraph are Gemini Enterprise Agent Platform features, the cloud product, not the local adk eval path. Locally the per rubric rationale above is the closest thing to that, and it is most of the value.

Supporting files

• scripts/eval_report.py, run any agent and evalset and print the judge's per rubric reasons. python eval_report.py AGENT_DIR EVALSET --config CONFIG.
• scripts/episode_finder/, the tiny verified demo agent, a helper with one find_episode tool.
• scripts/tests/eval/evalsets/episode_finder.evalset.json, a two case evalset.
• scripts/tests/eval/eval_config.json, local metrics only.
• scripts/tests/eval/eval_config_judge.json, adds the AI judge final_response_match_v2.
• scripts/tests/eval/eval_config_fixed.json, tool trajectory plus the judge, all green.
• scripts/tests/eval/eval_config_rubric.json, rubric judge with three rules, the reasons come from here.
• scripts/optimize/, the optimize and iterate demo. finder/ is the ungrounded agent that hallucinates, finder_fixed/ is the grounded fix, grounding.evalset.json asks about uncovered topics, eval_config_grounding.json is the grounding rubric that catches the made-up episodes, and propose_fix.py reads the judge's reason and proposes the instruction fix (a model, not the judge, writes the proposal). python optimize/propose_fix.py AGENT_DIR EVALSET --config CONFIG.

Documentation Pages

You MUST fetch the matching page below before writing eval code. These hosted docs are the source of truth for the metric names, config schema, and thresholds, do not rely solely on the examples above.

• https://github.com/google/agents-cli
• https://github.com/google/skills
• https://developers.googleblog.com/driving-the-agent-quality-flywheel-from-your-coding-agent/

From the episode

Video, link once it is up.