suite-converter

Suite Converter

Converts downloaded eval datasets into valid Margin Eval test suites.

Purpose

Use this skill when converting an external eval dataset into the Margin Eval suite format.

The job has three parts:

• map the source format into Margin's filesystem contract
• preserve the source verifier's intended behavior without breaking Margin's execution model
• validate the conversion on a small sample with margin run before scaling up

Target Margin Contract

<suite-name>/
├── suite.toml
└── cases/
    └── <case-name>/
        ├── case.toml
        ├── prompt.md
        ├── env/
        │   └── Dockerfile
        ├── tests/
        │   ├── test.sh
        │   └── <other test files>
        └── oracle/              # optional
            └── solve.sh

suite.toml

kind = "test_suite"
name = "<suite-name>"
description = "<description>"
cases = [
  "<case-1>",
  "<case-2>",
]

The cases array lists directory names under cases/, in the order they should run.

case.toml

kind = "test_case"
name = "<case-name>"
description = "<one-line description>"
agent_cwd = "/"
test_cwd = "/"
test_timeout_seconds = 1800

[metadata]
difficulty = "easy"
category = "programming"
tags = ["tag1", "tag2"]

[metadata] can be preserved for human context and future tooling, but the current Margin compiler/runtime primarily cares about the execution fields such as kind, name, description, image, agent_cwd, test_cwd, and test_timeout_seconds.

Key rules:

• name must match its directory name exactly
• kind is always "test_case"
• test_timeout_seconds is an integer (seconds)
• agent_cwd is the directory where the agent is expected to start and do its work inside the container
• test_cwd is the working directory where test.sh runs inside the container

Image handling, exactly one of:

• image = "registry/repo@sha256:<64hex>" for a pre-built, digest-pinned image
• Omit image and place a Dockerfile at env/Dockerfile to build at compile time

prompt.md

The full task description sent to the agent as its initial prompt. Must not be empty. Copy the source's instruction/prompt file as-is — don't summarize or reformat it.

tests/test.sh

The grading script. This is the evaluator. There is no separate grader abstraction.

env/Dockerfile

Container environment. Can include supporting files alongside the Dockerfile. The entire env/ directory is the build context.

oracle/solve.sh

Optional reference solution. Not executed during normal eval runs. In the current Margin implementation, oracle/ is informational only and is not part of the active compile/runtime path.

Conversion Invariants

These rules are cross-source and should hold for every conversion.

Verifier rules

• Must be executable (chmod +x)
• Exit 0 = pass, 1 = fail, 2 = infra
• Do not write reward.txt. The verifier process exit code is the authoritative result.
• All files in tests/ are packaged together and staged at {test_cwd}/tests/ in the container
• If the source suite uses absolute test-asset paths such as /tests/..., rewrite them to Margin-compatible paths such as tests/... or "${PWD}/tests/...".
• If the source verifier derives test-asset paths from environment variables, normalize or override those variables to Margin-compatible values. Do not rely only on fallback expressions when the source environment may already set an incompatible path such as /tests.
• Ensure the verifier creates any directories it expects before writing output artifacts.
• Use exactly one authoritative test location. If the verifier copies tests from tests/ into the workspace, its test command must avoid rediscovering the mounted tests/ tree. If it runs tests directly from tests/, do not also copy them into the workspace.
• Never leave a wrapper that always exits 0. Every verifier must terminate through explicit pass, fail, or infra paths.
• If the source suite expects status or report artifacts in addition to the exit code, preserve that behavior, but do not treat any single artifact format as universal.

Verdict policy

Use this attribution rule across conversions:

• pass: the harness reached a trustworthy verdict and the candidate satisfied the task
• fail: the harness reached a trustworthy verdict and the candidate did not satisfy the task
• infra: the harness could not reach a trustworthy verdict for reasons not attributable to the candidate

Classify these as fail:

• missing candidate-owned artifact
• candidate compile/import/build/runtime failure
• candidate timeout after candidate logic begins
• wrong output, failed assertions, malformed candidate-generated output
• candidate-caused repo state that prevents intended test execution

Classify these as infra:

• verifier/bootstrap/parser dependency failure
• suite-owned config or hidden test asset is missing or malformed
• verifier cannot interpret logs or required verifier artifacts are missing
• harness-side timeout before candidate evaluation meaningfully begins

If a parser or verifier layer cannot produce a trustworthy verdict, default to infra unless the harness already has direct evidence of a candidate-caused failure.

Case identity rules

• case.toml.name must match the case directory name exactly
• If source-name sanitization causes collisions, append a stable suffix such as part of the source task ID or UUID

Working directory rules

• agent_cwd and test_cwd are separate concepts:
  • agent_cwd is where the agent starts
  • test_cwd is where the verifier runs
• Set agent_cwd to the directory where the source task expects the agent to operate on the codebase or files
• Set test_cwd to the directory where the verifier should execute
• Do not assume they are the same. Only use the same value for both when the source task clearly indicates that the agent and verifier operate from the same directory

Validation rules

• Do not assume the conversion is correct until a sample run succeeds without harness-level issues
• Distinguish real task failures from conversion failures
• If the sample exposes harness issues, fix them and rerun the sample before scaling up
• Validate all three terminal states when practical:
  • a known-good sample yields 0
  • a known-bad or unsolved sample yields 1
  • an induced verifier/setup failure yields 2

Decision Rules

Inferring working directories

Parse the Dockerfile for WORKDIR directives. Use the last WORKDIR value found as the default working directory inside the container.

Use that information carefully:

• infer test_cwd from the verifier and Dockerfile execution context
• infer agent_cwd from where the source task expects the agent to work on the repo or files
• do not default agent_cwd to test_cwd unless the source task clearly uses the same directory for both

If no better signal exists:

• default test_cwd to the last WORKDIR, or "/" if none exists
• choose agent_cwd from the most plausible agent workspace for the task, rather than assuming it matches test_cwd

Clarifying Docker behavior

If the source suite provides more than one viable environment path and the user has not explicitly said which to use, ask before converting.

Common ambiguous cases:

• The source provides both a Dockerfile-like environment and a prebuilt image reference
• The source provides environment files that could be rebuilt, but the user may prefer to keep the original image
• The user may want you to rebuild the image, push it to a registry, and reference the new image in case.toml

When Docker behavior is ambiguous, ask which of these they want:

• Preserve the original image reference in image
• Convert using env/Dockerfile
• Rebuild and publish a new image, then use that in image

Do not modify Dockerfiles just to express pass/fail/infra policy unless the user explicitly asks for environment changes. Prefer verifier-level changes for verdict semantics.

Conversion Workflow

1. Identify the source format by inspecting the input directory (look for characteristic files like task.toml, case.toml, etc.)
2. Choose a suite name — derive from the dataset name or ask the user
3. Create the output structure: <suite-name>/cases/
4. For each task/case in the source, create a case directory and convert:
   • Config file → case.toml
   • Prompt/instruction → prompt.md
   • Test scripts → tests/
   • Dockerfile/environment → env/
   • Solution (if any) → oracle/
5. Generate suite.toml listing all case names (sorted alphabetically)
6. Validate: every case must have case.toml, prompt.md, tests/test.sh, and either image or env/Dockerfile
7. Set permissions: chmod +x on tests/test.sh and oracle/solve.sh
8. Ensure unique case names: if sanitization causes collisions, append a stable suffix (for example part of the source task ID/UUID)
9. Smoke test a sample first: convert a small sample of 2-5 representative cases before committing to the full dataset
10. Run Margin on that sample: execute the sample with margin run and carefully inspect the results, logs, and verifier behavior to identify conversion issues before scaling up
11. Validate verifier behavior: confirm the converted tests/test.sh locates its test assets under {test_cwd}/tests, returns a non-zero exit code when the underlying tests fail, and does not discover the same tests from both the workspace and tests/
12. Validate agent starting context: confirm agent_cwd points at the directory where the agent should actually begin work for the task
13. Fix and rerun before scaling up: if the sample exposes harness issues, adjust the conversion so the verifier has a single authoritative test location, then rerun the sample before converting the full dataset

Generated Wrapper Guidance

If the source has test scripts (e.g., pytest files) but no test.sh, generate a wrapper:

#!/bin/bash
set -euo pipefail

pass()  { printf 'VERDICT: PASS\n'; exit 0; }
fail()  { printf 'VERDICT: FAIL\n'; exit 1; }
infra() { printf 'VERDICT: INFRA\n' >&2; exit 2; }

<dependency installation commands or bootstrap checks>

set +e
<test runner command, e.g., pytest tests/test_outputs.py -rA>
exit_code=$?
set -e

case "$exit_code" in
  0) pass ;;
  1) fail ;;
  *) infra ;;
esac

Common failure signatures to look for during sample validation:

• Path mismatch: the verifier still points at /tests/... instead of {test_cwd}/tests/...
• Duplicate discovery: the same tests are collected from both the workspace and tests/
• Masked failures: the wrapper reaches an ambiguous state but still exits 0 or 1

Source-Specific Adapters

Harbor

See references/harbor-mapping.md for the complete field-by-field mapping.

Harbor datasets (downloaded via harbor datasets download) have this layout:

<output-dir>/
├── <uuid-1>/<task-name>/
│   ├── task.toml
│   ├── instruction.md
│   ├── environment/
│   │   ├── Dockerfile
│   │   └── <setup scripts...>
│   ├── tests/
│   │   ├── test.sh
│   │   └── test_*.py
│   └── solution/
│       └── solve.sh
├── <uuid-2>/<task-name>/
│   └── ...

Each UUID directory wraps exactly one named task subdirectory.

Harbor-specific steps

1. Scan the input directory for UUID subdirectories (each contains one task folder)
2. For each task: a. Read task.toml b. Use the inner directory name, not the UUID, as the case name c. Sanitize the case name to be filesystem-safe d. If sanitization collides with an existing case name, append a stable suffix derived from the Harbor UUID e. Create cases/<case-name>/ f. Generate case.toml per references/harbor-mapping.md g. Copy instruction.md to prompt.md h. If Harbor provides both a reusable image path and rebuildable environment files, and the user has not chosen a Docker strategy, stop and ask which behavior they want i. If task.toml declares [environment].docker_image, map it to Margin image when the user wants to preserve or reuse the source image j. If the user wants a Dockerfile-backed case, copy environment/ to env/ k. If the user wants a rebuilt and published image, build from environment/, publish to the selected registry, and write the published image reference to Margin image l. Copy tests/ to tests/ m. Apply the general verifier rules above, especially path normalization, environment-variable overrides, single test location, and exit-code propagation n. If solution/ exists, copy it to oracle/ as reference material only
3. Parse WORKDIR from env/Dockerfile to help infer working directories when using a Dockerfile-backed environment or rebuilding/publishing a new image. Use it directly for test_cwd when it matches the verifier context, and infer agent_cwd separately from the source task layout or expected repo workspace. If using a Harbor docker_image without a Dockerfile, infer both directories from the source task and verifier rather than assuming they match.
4. Generate suite.toml with all case names
5. chmod +x all .sh files in tests/ and oracle/

What to drop

These Harbor fields have no Margin Eval equivalent and are safely dropped:

• version (replaced by kind = "test_case")
• [environment].build_timeout_sec
• [environment].allow_internet
• [environment].mcp_servers
• [verifier.env], [solution.env]

What to preserve as metadata

Resource constraints are useful context even though Margin doesn't enforce them at the case level. Store them in [metadata] for documentation only:

[metadata]
# ... standard fields ...
harbor_cpus = 1
harbor_memory_mb = 2048
harbor_gpus = 0
harbor_agent_timeout_sec = 120

Validation Checklist

Before declaring a conversion complete, verify:

• Every case has case.toml, prompt.md, tests/test.sh, and either image or env/Dockerfile
• case.toml.name matches the directory name
• agent_cwd points at the directory where the agent should actually start
• test_cwd resolves to a real working directory assumption for the case
• Required shell scripts are executable
• The sample suite runs under margin run
• The verifier reads test assets from the correct location
• The verifier does not discover the same tests from both the workspace and tests/
• The verifier propagates real failures with a non-zero exit code
• Any remaining sample failures are real task failures, not harness failures