# Run Examples

> Run Examples is an Agent Skill for testing & quality published in the symfony/ai repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Picks and runs the examples under examples/ that exercise the code changed on the current branch / PR / diff, as a phase-two integration check after a code review. Use when the user says "run the relevant examples", "verify with examples", "test the changes with examples", "/run-examples", or asks to execute matching examples for a PR/branch. Always proposes the candidate list (with reasons + skip notes) and waits for approval before executing. Handles missing API keys via examples/.env.local and brings up the right Docker services when needed.

- Skill name: `run-examples`
- Category: [Testing & Quality](https://www.openparable.com/skill/testing-quality/1)
- Source: https://github.com/symfony/ai/blob/HEAD/.claude/skills/run-examples/SKILL.md
- Repository: symfony/ai (1,159 GitHub stars when collected)

## SKILL.md

# Run Examples

You help the maintainer of the Symfony AI monorepo run the subset of
`examples/` scripts that actually exercise the code on the current branch / PR.
This is a *phase-two* check that runs **after** a code review (e.g. via
`/pr-review`) once the maintainer is comfortable with the change itself and
wants to see the examples still light up green.

Two non-negotiable behaviors:

1. **Propose first, then execute.** Always present the candidate examples and
   wait for approval before running anything. The point of this skill is to
   give the maintainer a thinking step, not to autorun.
2. **Don't pretend skipped examples passed.** If an example can't run because a
   required key is missing from `examples/.env.local`, list it as *skipped*
   with the reason. Don't quietly drop it.

## 1. Determine the changeset

Same approach as `pr-review`:

- **PR number given** (e.g. `/run-examples 1234`): `gh pr checkout 1234`
  if not already on the branch, then derive the diff against the PR's
  `baseRefName` (`gh pr view 1234 --json baseRefName`).
- **No argument**: review the current branch against the merge-base with `main`
  (fall back to `master`).
- **Explicit base ref given** (e.g. the maintainer says "base is `develop`" or
  "compare against `<sha>`"): use that ref as the base instead of detecting
  one. This also covers replaying a historical change from a worktree.

Get the changed files with `git diff --name-only <base>..HEAD`. Read the actual
diff (`git diff <base>..HEAD`) only if you need the contents to make better
mapping decisions — for picking *which* examples to run, the file list is
usually enough.

## 2. Map changes → relevant examples

The mapping is mostly mechanical because the monorepo has strong conventions.
Apply these in order; a single change can match multiple categories.

### 2a. Bridge changes (most common, most direct)

`src/<component>/src/Bridge/<Vendor>/...` → examples that import that namespace.

The reliable way to find them is to grep `examples/` for the namespace prefix
of the changed Bridge:

```
git diff --name-only <base>..HEAD | grep '^src/.*/src/Bridge/'
# for each Bridge directory, grep examples/ for its namespace
grep -rl "Symfony\\\\AI\\\\<Component>\\\\Bridge\\\\<Vendor>" examples/
```

This catches both the "obvious" matches (Bridge `OpenAi/Embeddings/` → uses in
`examples/openai/embeddings.php`) and the cross-cutting ones (a Platform
OpenAI change is also exercised by `examples/rag/postgres.php` because it uses
OpenAI for embeddings).

Common shortcuts that hold:

- `src/platform/src/Bridge/<Vendor>/` → `examples/<vendor>/*.php` plus any
  `examples/rag/`, `examples/store/`, `examples/indexer/`, `examples/retriever/`,
  `examples/chat/`, `examples/memory/` script that uses that vendor for
  embeddings or chat.
- `src/store/src/Bridge/<Backend>/` → `examples/store/<backend>.php`,
  `examples/rag/<backend>.php` (and variants like `<backend>-openai.php`),
  plus indexer/retriever/memory scripts that import the bridge.
- `src/chat/src/Bridge/<Backend>/` → `examples/chat/persistent-chat-<backend>.php`.
- `src/agent/src/Bridge/<Tool>/` → `examples/toolbox/<tool>.php` and any
  example that uses that tool.

When in doubt, trust the grep result over the path heuristic.

**Express runs at folder granularity when you can.** If most/all candidates
live in one subdirectory (typical for a Bridge change touching only
`<vendor>/` scripts), propose running that subdirectory via the runner —
`./runner <vendor>` — rather than enumerating each script. The runner is the
intended unit; it parallelises, captures output, and reports a summary table.
Use `./runner --filter=<pattern>` when you want a name-based subset
(e.g. `--filter=toolcall` to hit toolcall variants across vendors). Reserve
per-script enumeration for when the candidates are scattered across
directories or you genuinely want only a handful from a much larger set.

### 2b. Component-level changes (Platform, Agent, Store, Chat core)

Changes outside any Bridge folder (e.g. `src/platform/src/Result/`,
`src/agent/src/Toolbox/`, `src/store/src/Document/`) tend to affect many
examples. Don't enumerate hundreds of scripts. Instead:

- Identify which *capabilities* the change touches (streaming, structured
  output, tool calling, message bag, vectorizer, ...).
- Pick a small representative set of examples per affected capability —
  ideally **one per capability**, two if the capability has meaningfully
  different shapes (e.g. tool-call sync vs. tool-call streaming). The
  maintainer can always ask to broaden; reining in is harder.
- **Prefer Docker-free examples.** Unless the change is specifically about a
  store / persistence backend, pick representatives that don't need
  `docker compose up`. Persistent-chat → prefer `cache` / `doctrine-dbal` /
  `session` / `static` over `redis` / `mongodb` / `meilisearch`. RAG → prefer
  `in-memory.php` over `postgres.php`. Skip the Docker spin-up cost when it
  doesn't add coverage for the diff.
- If the change is broad and uncontroversial (refactor, type tightening,
  internal rename), suggest running `./runner` against one or two vendor
  subdirectories the maintainer is most familiar with rather than hand-picking.

When uncertain, propose fewer examples and explicitly say so — the maintainer
can ask for more.

### 2c. When nothing in `examples/` covers the diff

Some diffs are simply not within scope of this skill. The clear cases:

- **Bundle-only diffs** (`src/ai-bundle/`, `src/mcp-bundle/`) — `examples/`
  scripts instantiate factories directly and never boot the Symfony kernel,
  so they don't exercise bundle wiring at all.
- **Fixtures, docs, infra, CI** — nothing to run.
- **Tests-only diffs** — the change *is* the test; running examples adds no
  signal.

In these cases the output is short and final:

1. State that nothing in `examples/` applies, with a one-line reason.
2. Optionally add a "Gap check" line *only* if the diff adds a new public
   capability that plausibly should have a demonstrating example (a new
   bridge, a new content type, a new public API surface) — but not for
   internal refactors, tests, or wiring fixes.
3. Stop.

**Do not** suggest running phpunit, the demo app, CI workflows, or any other
tool — those have their own coverage and are out of this skill's scope.
**Do not** propose a "smoke test of a random vendor anyway" as a fallback —
if the diff doesn't touch that vendor's code, running it gives no signal.
**Do not** add a "Recommendation" section pointing at non-`examples/`
tooling. Saying "nothing applies" with a reason *is* the complete answer —
trust the maintainer to know what their other tools are.

## 3. Filter by runnability

For every candidate example produced in step 2, decide whether it can actually
run **on this machine right now**. Each example reaches this state by passing
through:

### 3a. Required env vars

`examples/bootstrap.php` defines `env(string $var)` that prints an error and
`exit(1)` if the var is empty. So the gate is: every `env('FOO')` call inside
the example file must have `FOO` non-empty in `examples/.env.local` (or, as a
fallback, in the shell environment / `examples/.env` defaults).

To check:

```
grep -oE "env\('[^']+'\)" path/to/example.php | sort -u
```

Then look up each var in `examples/.env.local` (parse it: `KEY=VALUE`, ignore
blank lines and `#` comments, treat empty `VALUE` as missing). `examples/.env`
holds defaults for *infrastructure* env vars (hosts, ports for local Docker
services); `.env.local` is where the maintainer keeps API keys. A var is
"available" if it's filled in either file.

### 3b. Docker-backed services

Some examples talk to a service from `examples/compose.yaml` rather than (or
in addition to) a third-party API. Detection heuristic: if the example
references any of these env vars, it needs Docker:

| Env var(s) used                                  | Compose service name |
|--------------------------------------------------|----------------------|
| `MARIADB_URI`                                    | `mariadb`            |
| `POSTGRES_URI`                                   | `postgres`           |
| `MONGODB_URI`                                    | `mongodb`            |
| `REDIS_HOST`                                     | `redis`              |
| `MEILISEARCH_HOST`                               | `meilisearch`        |
| `QDRANT_HOST`                                    | `qdrant`             |
| `WEAVIATE_HOST`                                  | `weaviate`           |
| `MILVUS_HOST`                                    | `milvus` (+ `etcd`, `minio`) |
| `ELASTICSEARCH_ENDPOINT`                         | `elasticsearch`      |
| `OPENSEARCH_ENDPOINT`                            | `opensearch`         |
| `CHROMADB_HOST` / `CHROMADB_PORT`                | `chromadb`           |
| `TYPESENSE_HOST`                                 | `typesense`          |
| `NEO4J_HOST` / `NEO4J_PASSWORD`                  | `neo4j`              |
| `SURREALDB_HOST`                                 | `surrealdb`          |
| `MANTICORESEARCH_HOST`                           | `manticore`          |
| `CLICKHOUSE_HOST`                                | `clickhouse`         |
| `POGOCACHE_HOST`                                 | `pogocache`          |
| `PINECONE_HOST` (when set to `127.0.0.1`/local)  | `pinecone`           |
| `LITELLM_HOST_URL`                               | `litellm` (+ `litellm-db`) |

Examples that hit a *remote* API (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`,
`AZURE_*`, `AWS_*`, `BEDROCK`, `OPENROUTER_KEY`, ...) do not need Docker —
they only need the key.

The local model runners (`OLLAMA_HOST_URL`, `LMSTUDIO_HOST_URL`,
`DOCKER_MODEL_RUNNER_HOST_URL`) point at a *separately running* local server
that the maintainer manages outside `compose.yaml`. The defaults in
`examples/.env` already point them at sensible local URLs, so treat them
like any other env-gated example: if the host var is set, the example is
runnable. If the local server isn't actually up, the example will exit
early and the runner will report it as Skipped — that's fine. Don't
preemptively mark these as unrunnable just because they need a local
runner; the maintainer often has one running.

### 3c. Runnability outcome

For each candidate, classify into one of:

- **runnable** — all required env vars are set; any Docker services it needs
  are listed for confirmation.
- **skipped (missing keys)** — list which env vars are empty so the maintainer
  can decide whether to fill them in.
- **needs Docker** — runnable in principle but requires `docker compose up -d --wait <services>`
  first. Group these so we can start everything in one command.

## 4. Propose the plan

Output a structured proposal. Don't run anything yet. Express the run as
**runner invocations** when possible — `./runner <subdir>` or `./runner
--filter=<pattern>` — and only fall back to per-script enumeration when the
candidates don't sit naturally inside one folder.

```
## Proposed example runs for <branch / PR #>

Changeset: <one-line summary — e.g., "Adds streaming support to
src/platform/src/Bridge/OpenAi/Embeddings/">

### Will run
`cd examples && ./runner openai`  — all 17 OpenAI scripts, exercises the
   modified `OpenAi\Embeddings\ResultConverter`

If multiple subdirs are involved, list each on its own line. If the right
unit is "a few specific scripts", list those instead (with a one-line "why"
each).

### Docker services to start first
`docker compose up -d --wait postgres`  — needed by `rag/postgres.php` only

(Omit the Docker section entirely if no service is required.)

### Skipped (M)
- `examples/anthropic/*` — `ANTHROPIC_API_KEY` is empty in `.env.local`

Reply with "go" to run, or tell me which to keep / drop.
```

Keep it tight. Be selective rather than maximalist — propose the *minimum*
that meaningfully exercises the diff, not the maximum that touches the
namespace. If a folder has 14 scripts and only 2 are relevant to the change,
list those 2 — don't propose `./runner <vendor>` just because you can. The
runner shortcut is for when running the whole folder is genuinely the right
unit (e.g. a Bridge-internal change that could affect any of its examples).

## 5. Execute (after approval)

Once the maintainer says go (or edits the list):

1. **Start Docker services** if any are needed:
   ```
   docker compose up -d --wait <service> <service> ...
   ```
   Run this from `examples/`. `--wait` blocks until healthchecks pass.
   Don't blanket-up the entire `compose.yaml` — only the services we need.
   If services fail to come up, stop and surface the error; don't try to
   run examples against a half-broken stack.

2. **Run the examples** via `./runner` by default — it parallelises and
   reports a Finished / Failed / Skipped summary table:
   ```
   cd examples && ./runner <subdir>           # whole subdir, e.g. ./runner openai
   cd examples && ./runner <a> <b> <c>        # multiple subdirs
   cd examples && ./runner --filter=<pattern> # by name pattern, e.g. --filter=toolcall
   ```
   "Skipped" rows mean the example exited early — usually a missing env
   var; cross-reference with the step-3 skip list. Only fall back to a
   bare `php <path>` invocation when you specifically need the verbose
   output of one example (e.g. to debug a failure with `-vvv`).

3. **Report results**. Group by outcome:
   - Failures: show the example path, exit code, and a short snippet of the
     error output (stderr if available, otherwise stdout). Don't paste
     hundreds of lines — quote the first few lines that show the actual
     error.
   - Successes: just count them.
   - Skipped: pass through what was already known.

   If anything failed, suggest re-running with `-vvv` for full HTTP logs:
   `php <path> -vvv`.

4. **Don't tear down Docker.** Leave services up — the maintainer probably
   wants to iterate. Only suggest `docker compose down` if they ask.

## Principles

- **Be selective, not maximalist.** Better to propose 5 highly-relevant
  examples than 50 vaguely-related ones. "It imports the namespace" alone
  isn't a reason to run something — the diff has to plausibly affect what
  the example exercises. Reining in is harder than expanding.
- **Prefer the runner over per-script lists** when the unit of work is
  "a folder" or "a name pattern". `./runner openai` reads better than 17
  bullet points. Only enumerate scripts when the candidates are scattered
  or genuinely a small subset.
- **Prefer Docker-free representatives** unless the diff is specifically
  about a store / persistence backend. Don't make the maintainer spin up
  containers just to verify a change in core platform/agent code.
- **Explain your reasoning per example.** A one-line "why" next to each
  candidate (which file/namespace it ties to) lets the maintainer veto
  quickly.
- **Surface skips, don't hide them.** If half the candidates can't run,
  that's information — say so up front so the maintainer can decide whether
  to fill in keys before running.
- **Don't auto-fix env vars.** Never write to `.env.local` or suggest
  values for missing API keys. Just report what's missing.
- **Stay in scope.** This skill is *only* about `examples/`. If the diff
  isn't covered by any example, say so and stop — don't pivot to phpunit,
  demo runs, or other tools that have their own CI coverage.

---

Source: https://www.openparable.com/skill/testing-quality/run-examples (OpenParable). Cite the canonical URL above.
