OpenParable
Skip to content
← Back to skills

Robustmq Chaos Test

Testing & Quality
robustmq-chaos-test

Robustmq Chaos Test is an Agent Skill for testing & quality published in the robustmq/robustmq repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

7×24 chaos testing for RobustMQ. Injects broker-kill and network-delay faults, validates SDK client resilience across Python/Go/Rust/Java, and publishes a Markdown + JSON report to GitHub after each run.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/robustmq/robustmq/blob/HEAD/chaos-test/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/robustmq-chaos-test, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

RobustMQ Chaos Test Skill

When to Use

Cron trigger (every 4 hours): the system message will say

"按 P0 跑一轮 RobustMQ 故障场景"

Manual CLI: user says something like

"帮我跑一轮 RobustMQ chaos 测试" / "run a chaos test round"

In both cases execute the Full Run below.

If the user says "按 P1 跑一轮" or names a specific scenario, execute the Single Scenario flow for that scenario only.


Pre-check

Before starting any run:

  1. Call cluster_manage(action=status).
    • If status is NOT stopped, call cluster_manage(action=stop) to clear any leftover state from a previous run.
  2. Verify chaos-test/config.yml has cluster.binary and cluster.project_root filled in correctly (the cluster tool fails fast if binary is missing — surface that error immediately and stop).

Scenario Catalogue

Scenario namefault_typetargetparamsCore?
broker-kill-singlebroker-killrobustmq-server—✅
network-delay-100msnetwork-delayeth0delay_ms=100, jitter_ms=10—
leader-transferbroker-killrobustmq-server—✅

Note: Update this table when new scenarios are added. Target names and interface names depend on the deployment environment — verify before running.

Core scenarios: broker-kill-single, leader-transfer. Run passed = all core pass AND non-core pass rate ≥ 75%.


Single Scenario — 5-Step Flow

Execute these steps sequentially. Do NOT skip steps.

Step 1 — Baseline Snapshot

observability(action=snapshot, data_dirs=<from cluster start>)

Record the snapshot as baseline. Proceed even if some metrics are unavailable; log a warning but do not abort.

Step 2 — Inject Fault

chaos(action=inject, fault_type=<type>, target=<target>, params=<params>)

Save the returned fault_id. If inject returns an error, mark the scenario passed=False with status=inject_error and skip to Step 5 (skip recover).

Step 3 — Fault-Period SDK Observation (record only)

client(action=run, scenario=<scenario>, cluster_endpoint=<endpoint>)

Record all results. Do NOT use these results to determine pass/fail. Their only purpose is observability — they show what clients experienced during the fault. A high loss rate here is expected and normal.

Step 4 — Recover

chaos(action=recover, fault_id=<fault_id>)

If recover returns an error, log it and continue — attempt self-healing validation anyway.

Step 5 — Self-Healing Validation (sole pass/fail basis)

Wait 60 seconds after recovery, then run:

client(action=run, scenario=<scenario>, cluster_endpoint=<endpoint>)

Pass criteria (ALL must hold):

  • exit_code == 0
  • lost == 0
  • p99_ms < 500

If any criterion fails → scenario passed=False. If status=script_format_error → scenario passed=False, note the format error separately (this is a test-infrastructure issue, not a RobustMQ bug).


Full Run Flow

  1. Pre-check (see above).
  2. Start cluster: cluster_manage(action=start) → save endpoint and data_dirs.
  3. Run each scenario using the Single Scenario flow.
    • Run scenarios sequentially (not in parallel) to avoid interference.
    • If a scenario crashes the cluster (all brokers dead), restart it before continuing: cluster_manage(action=stop) → cluster_manage(action=start).
  4. Stop cluster: cluster_manage(action=stop).
  5. Generate report: report(action=generate_and_push, run_data={...}).
    • run_data must include: run_id, started_at, finished_at, scenarios.
    • Each scenario entry: scenario name, sdk, passed, sent, received, lost, p99_ms, duration_seconds, errors, status.
  6. Send Feishu notification:
    • If run_passed=True: send brief pass message with github_url.
    • If run_passed=False: send failure alert listing failed scenarios and github_url.
    • If consecutive_failures >= 3: prepend 🚨 连续 {n} 轮失败,请人工介入.

Circuit Breaker

Track consecutive_failures across runs (persist in your memory or state):

  • Increment on run_passed=False.
  • Reset to 0 on run_passed=True.
  • If consecutive_failures >= 3: send an urgent Feishu alert and pause the cron schedule. Do NOT continue running automatically until a human acknowledges and resets the counter.

Feishu Message Templates

Pass:

✅ RobustMQ 故障测试通过
Run ID: {run_id}  时间: {finished_at}
核心场景: 全部通过  总通过率: {pass_rate}%
报告: {github_url}

Fail:

❌ RobustMQ 故障测试失败
Run ID: {run_id}  时间: {finished_at}
失败场景: {failed_scenario_list}
报告: {github_url}

Circuit breaker:

🚨 连续 {consecutive_failures} 轮失败,请人工介入
最后失败: {run_id}  {finished_at}
报告: {github_url}

Pitfalls

  • Never judge pass/fail on fault-period results (Step 3). Only Step 5 post-recovery validation counts.
  • script_format_error ≠ RobustMQ bug. Report it separately; do not inflate the failure count. Fix the script first.
  • ROBUSTMQ_HOME must be set before any run. The cluster tool returns an error immediately if it is not — surface it and stop rather than retrying.
  • Consecutive failures count whole runs, not individual scenarios. One run with two failed scenarios = 1 failure, not 2.
  • eth0 is not universal. The network-delay target interface name varies by host. Verify it before running in a new environment.
  • Deploy Key permissions. If report returns push_error, the reports are still written locally at json_path / markdown_path. Investigate the key before declaring the run lost.
  • 60-second wait is mandatory. Do not skip or shorten it — RobustMQ leader election and connection re-establishment take time.