# Swe Bench

> Swe Bench is an Agent Skill for testing & quality published in the opensquilla/opensquilla repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Run SWE-bench instances with an OpenSquilla agent inside the official Docker images. Trigger when the user wants to run/solve/evaluate a SWE-bench instance (e.g. 'run django__django-16429', 'test OpenSquilla on SWE-bench', '跑一道 SWE-bench 题'), benchmark the agent on SWE-bench_Verified or SWE-bench_Multilingual, or check whether a generated patch resolves an instance. Optional dependency — install via `pip install opensquilla[swebench]`; also needs the docker CLI and an OPENROUTER_API_KEY.

- Skill name: `swe-bench`
- Category: [Testing & Quality](https://www.openparable.com/skill/testing-quality/1)
- Source: https://github.com/opensquilla/opensquilla/blob/HEAD/src/opensquilla/skills/bundled/swe-bench/SKILL.md
- Repository: opensquilla/opensquilla (6,081 GitHub stars when collected)

## SKILL.md

# swe-bench

Run a SWE-bench instance end-to-end: ensure the instance's Docker image is
available (pull from Docker Hub if missing), start a container, run an
OpenSquilla agent against the issue, collect the patch, and optionally run
the official evaluation.

## Prerequisites — guide the user, don't dead-end

SWE-bench mode runs the official evaluation images, so it needs the **Docker
CLI**. Docker is NOT a hard gate on this skill: if it is missing, the
`opensquilla swebench` command prints the exact install command for the
user's OS and exits. When that happens, relay the install guidance to the
user ("SWE-bench needs Docker — install it with `...`, then I can run this")
instead of saying the task is impossible. Also mention that solving a
**real-repository** coding task (not a benchmark instance) does NOT need
Docker — that is what the `code-task` skill is for.

## Commands

Solve one instance (auto-pulls the image when missing):

```
opensquilla swebench solve <instance_id> --dataset verified --json
```

- `--dataset` accepts `verified`, `multilingual`, or a full HuggingFace
  dataset name.
- Add `--evaluate` to run the official harness afterwards and report
  whether the patch actually resolves the instance (`resolved` in the
  JSON output).
- Add `--model <model>` / `--thinking <level>` to pin a model; leave them
  off to let squilla_router decide.
- `--timeout <seconds>` defaults to 1200.

Pre-fetch an image only:

```
opensquilla swebench pull <instance_id>
```

Evaluate an existing predictions file:

```
opensquilla swebench eval <predictions.jsonl> --dataset verified
```

## Reading the result

`solve --json` prints one JSON object: `state` (`patch_collected` is
success), `patch_path`, `artifact_dir`, `resolved` (true/false, or null
when `--evaluate` was not used), `duration_seconds`, `usage`
(cost/tokens), `error`.

The full artifact trail (prompt, agent log, transcript, usage, patch)
lives under `artifact_dir`.

## What to tell the user

1. Before starting: if the image is not local yet, warn that the first
   run pulls 1-3 GB and can take a few minutes; a solve typically takes
   10-30 minutes depending on the instance and timeout.
2. After finishing: report `state`, whether the patch is non-empty,
   `resolved` when evaluation ran, the cost from `usage`, and the
   `patch_path` so the user can inspect the diff.
3. On `failed`/`timeout`: quote the `error` field and point at
   `<artifact_dir>/agent_stdout.log` for the full trace.

## Constraints

- Images are x86_64; on ARM hosts only locally pre-built images work.
- The run happens on this machine (the gateway host) — docker must be
  reachable from here, and disk usage grows with each pulled image.
- Long runs block the command; for batches beyond a handful of
  instances, suggest the user run the CLI directly in tmux instead of
  going through chat.

---

Source: https://www.openparable.com/skill/testing-quality/swe-bench (OpenParable). Cite the canonical URL above.
