OpenParable
Skip to content
← Back to skills

Run Benchmark

Testing & Quality
run-benchmark

Run Benchmark is an Agent Skill for testing & quality published in the AgibotTech/genie_sim repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Launch a geniesim_benchmark task locally (typically inside the GUI Docker container) against a user-provided inference server, using the `geniesim benchmark run` CLI verb. Trigger: When the user asks to "run geniesim", "本地跑仿真", "启动仿真任务", "run a benchmark", "launch <some>_<config>.yaml", or wants to execute a benchmark task config (anything under `geniesim_benchmark/config/*.yaml`) against a remote inference host (ip:port).

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/AgibotTech/genie_sim/blob/HEAD/source/geniesim_benchmark/skills/run-benchmark/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/run-benchmark, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

When to Use

  • User wants to run a geniesim_benchmark task on their workstation (not the Challenge platform).
  • User has an inference server already running somewhere reachable and provides its ip:port.
  • User references any task config under source/geniesim_benchmark/src/geniesim_benchmark/config/.

Do not use for:

  • Submitting jobs to the Challenge platform → challenge-submit-job.
  • Verifying an inference server is healthy → use the check-inference skill.
  • Adding a new benchmark task → add-benchmark-task.

Critical Patterns

  1. Always collect three required inputs first:
    • Task config (basename, full path, or substring — the CLI resolves all three).
    • Inference IP.
    • Inference port.
  2. The runtime needs omni_python / Isaac Sim on the host. Inside the Genie Sim Docker image (geniesim docker up → geniesim docker into), that's already the case. Outside the container the user needs Isaac Sim installed system-wide.
  3. Working directory: anywhere under the repo root works — the CLI walks up to find scripts/ and uses find_spec to locate the benchmark package.
  4. Confirm before launching. The task spawns a full simulator and typically holds a GPU; ask before kicking it off if there's any ambiguity.

Workflow

Step 1 — Collect inputs

If the user hasn't named a config, list candidates:

geniesim benchmark categories      # show category counts
geniesim benchmark robots          # show robot counts
geniesim benchmark list --robot=<R> --category=<C>

Then ask via AskUserQuestion:

  • Task config: free-text (use the basename — e.g. g2op_if_pick_block_color).
  • Inference host as ip:port.

Step 2 — Probe inference (optional but recommended)

Before sinking minutes into Isaac Sim startup, sanity-check the server (uses the bundled corobot payload):

geniesim benchmark check-inference --infer-host=<IP>:<PORT>

See the check-inference skill to override the payload.

Step 3 — Run the task

Inside the GUI container (geniesim docker into):

geniesim benchmark run <CONFIG> --infer-host=<IP>:<PORT>

Example:

geniesim benchmark run g2op_if_pick_block_color --infer-host=<IP>:<PORT>

Step 4 — Pass-through overrides (when asked)

geniesim benchmark run forwards any unknown --key=value to the benchmark's ParameterServer. Common ones:

FlagMeaning
--app.headless=trueNo GUI (required on remote / batch hosts)
--benchmark.num_episode=NOverride episode count
--benchmark.seed=NRNG / instance-sampling seed
--benchmark.record=truePersist episode logs to output_dir
--benchmark.policy_class=…Use a different policy class

Full schema: source/geniesim_benchmark/src/geniesim_benchmark/config/params.py.

Commands (copy-paste summary for the user)

# Host — start the container (GUI by default; add --headless on remote/batch hosts)
cd /path/to/main
geniesim docker up

# Host — drop into a shell inside the running container
geniesim docker into
# inside container:
geniesim status                                       # verify the stack is healthy
geniesim benchmark check-inference --infer-host=<IP>:<PORT>
geniesim benchmark run <CONFIG> --infer-host=<IP>:<PORT>

Notes

  • If geniesim isn't on $PATH (the launcher wasn't installed), substitute python3 -m geniesim_cli benchmark … — same args, same behaviour.
  • <CONFIG> accepts the bare basename (g2op_if_pick_block_color), a full path, or a unique substring.
  • For batch evaluations, prefer geniesim benchmark batch --category=… --robot=… over a shell loop — it forwards extras consistently and prints a per-config pass/fail summary.
  • The new CLI replaces the older ad-hoc omni_python app/app.py --config … invocation. The new form normalizes interpreter selection, host shorthand, and config resolution.