OpenParable
Skip to content
← Back to skills

Check Inference

Testing & Quality
check-inference

Check Inference is an Agent Skill for testing & quality published in the AgibotTech/genie_sim repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Probe a model inference WebSocket server (e.g. `serve_policy`) and validate the response — using the `geniesim benchmark check-inference` CLI verb, which wraps the benchmark package's `check_inference.py`. Trigger: When the user asks to "check inference", "校验模型推理", "test inference server", "verify policy server", "ping the model", or provides an IP/port and wants to confirm a serve_policy / WebSocket inference server is working before running benchmarks.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/AgibotTech/genie_sim/blob/HEAD/source/geniesim_benchmark/skills/check-inference/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/check-inference, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

When to Use

  • Sanity-check whether a running inference server actually accepts requests and returns valid actions, before launching a full task.
  • User provides ip:port and a payload, and asks to verify connectivity / output validity.
  • Quick smoke test in CI / pre-deploy.
  • Diagnosing "the benchmark hangs / outputs garbage" — this probe surfaces protocol mismatches and NaN/Inf in actions before you sink time into a full simulator launch.

Do not use for:

  • Running the benchmark itself → use the run-benchmark skill.
  • Submitting jobs to the Challenge platform → challenge-submit-job.

What This Skill Does

Sends a saved corobot .pkl payload to ws://<HOST>:<PORT> and validates the reply:

  1. Loads the payload.
  2. Connects to the WebSocket server using msgpack-numpy.
  3. Sends one request, receives one action chunk.
  4. Validates: schema (key presence), per-dim min/max/mean/std, NaN/Inf flags, out-of-range checks (kind-aware: JOINT_ABS in radians, EEF_ABS in meters+quat, gripper in [0,1]), large jumps from the input state.

The payload is a corobot JSON-RPC envelope — {"method": "infer", "params": {...}} — and the server replies with {"result": {"left_arm": …, "right_arm": …, …}} (or {"error": …}).

Required Input

The user must provide:

  • HOST — server IP (e.g. 127.0.0.1).
  • PORT — server port (e.g. 8999).

PAYLOAD is optional — it defaults to the bundled corobot_payload.pkl. Pass a path only to override it (see Generating a payload below). If host/port are missing, ask the user before running.

How to Run

# Bundled payload — just point it at the server
geniesim benchmark check-inference --infer-host=<HOST>:<PORT>

# Override with your own payload (positional)
geniesim benchmark check-inference debug_preview/debug_0001.pkl \
    --host <HOST> --port <PORT>

If geniesim isn't on $PATH (the launcher wasn't installed), substitute python3 -m geniesim_cli benchmark check-inference … — same args, same behaviour.

Optional flags (forwarded to check_inference.py)

FlagEffect
--iters NSend N consecutive requests (default 1). Use 5–10 to catch flakiness.
--max-dims NMax idx rows printed per array (default 64).

Generating a payload

A canonical corobot_payload.pkl ships next to the script and is used by default, so you usually don't need to supply one. To probe with a fresh / task-specific observation, run a benchmark task with the corobot policy's debug dump enabled — it writes debug_preview/debug_NNNN.pkl (a {"payload": …, "obs": …} wrapper the probe unwraps automatically) — then pass that path.

Interpreting Output

The script prints structured sections:

SectionWhat it tells you
📦 PayloadPayload loaded and recognised as corobot.
🔌 ConnectingConnected to ws server (or refused).
📡 …responseServer returned a reply, schema check.
📊 <key>Per-dim min/max/mean/std + flags, per output (left_arm/right_arm/…).
⏱ latencyRound-trip latency.
OutcomeMeaning
✅ PASS — …Server is up and returning sane actions.
❌ Connection refused / timed outServer isn't listening on that host:port. Verify it's running and the firewall is open.
❌ … NaN / Inf …Server responded but model output is broken. Check the policy checkpoint and normalization stats — not a network problem.
❌ response missing 'result' dictServer schema mismatch — it isn't speaking the corobot JSON-RPC protocol the probe expects.
❌ server error: …The server returned a JSON-RPC error; read the message.
⚠️ OOB[…] flagsAction is finite but outside the kind's expected range. Could be a units bug (radians vs degrees) or an unnormalized output.

Dependencies

The script needs (in the Python env that runs python3):

  • msgpack
  • numpy
  • websockets

It does not need Isaac Sim — pure-Python deps only. The CLI deliberately uses python3 instead of omni_python here so the probe is snappy.

Resources

  • Script source: source/geniesim_benchmark/src/geniesim_benchmark/scripts/check_inference.py (resolved via the geniesim_benchmark package)
  • CLI dispatcher: source/geniesim_cli/src/geniesim_cli/commands/benchmark.py (_do_check_inference)
  • Payload dump hook: source/geniesim_benchmark/src/geniesim_benchmark/benchmark/policy/corobotpolicy.py