robustmq-chaos-test

RobustMQ Chaos Test Skill

When to Use

Cron trigger (every 4 hours): the system message will say

"按 P0 跑一轮 RobustMQ 故障场景"

Manual CLI: user says something like

"帮我跑一轮 RobustMQ chaos 测试" / "run a chaos test round"

In both cases execute the Full Run below.

If the user says "按 P1 跑一轮" or names a specific scenario, execute the Single Scenario flow for that scenario only.

---

Pre-check

Before starting any run:

1. Call cluster_manage(action=status).
   • If status is NOT stopped, call cluster_manage(action=stop) to clear any leftover state from a previous run.
2. Verify chaos-test/config.yml has cluster.binary and cluster.project_root filled in correctly (the cluster tool fails fast if binary is missing — surface that error immediately and stop).

---

Scenario Catalogue

Note: Update this table when new scenarios are added. Target names and interface names depend on the deployment environment — verify before running.

Core scenarios: broker-kill-single, leader-transfer. Run passed = all core pass AND non-core pass rate ≥ 75%.

---

Single Scenario — 5-Step Flow

Execute these steps sequentially. Do NOT skip steps.

Step 1 — Baseline Snapshot

observability(action=snapshot, data_dirs=<from cluster start>)

Record the snapshot as baseline. Proceed even if some metrics are unavailable; log a warning but do not abort.

Step 2 — Inject Fault

chaos(action=inject, fault_type=<type>, target=<target>, params=<params>)

Save the returned fault_id. If inject returns an error, mark the scenario passed=False with status=inject_error and skip to Step 5 (skip recover).

Step 3 — Fault-Period SDK Observation (record only)

client(action=run, scenario=<scenario>, cluster_endpoint=<endpoint>)

Record all results. Do NOT use these results to determine pass/fail. Their only purpose is observability — they show what clients experienced during the fault. A high loss rate here is expected and normal.

Step 4 — Recover

chaos(action=recover, fault_id=<fault_id>)

If recover returns an error, log it and continue — attempt self-healing validation anyway.

Step 5 — Self-Healing Validation (sole pass/fail basis)

Wait 60 seconds after recovery, then run:

client(action=run, scenario=<scenario>, cluster_endpoint=<endpoint>)

Pass criteria (ALL must hold):

• exit_code == 0
• lost == 0
• p99_ms < 500

If any criterion fails → scenario passed=False. If status=script_format_error → scenario passed=False, note the format error separately (this is a test-infrastructure issue, not a RobustMQ bug).

---

Full Run Flow

1. Pre-check (see above).
2. Start cluster: cluster_manage(action=start) → save endpoint and data_dirs.
3. Run each scenario using the Single Scenario flow.
   • Run scenarios sequentially (not in parallel) to avoid interference.
   • If a scenario crashes the cluster (all brokers dead), restart it before continuing: cluster_manage(action=stop) → cluster_manage(action=start).
4. Stop cluster: cluster_manage(action=stop).
5. Generate report: report(action=generate_and_push, run_data={...}).
   • run_data must include: run_id, started_at, finished_at, scenarios.
   • Each scenario entry: scenario name, sdk, passed, sent, received, lost, p99_ms, duration_seconds, errors, status.
6. Send Feishu notification:
   • If run_passed=True: send brief pass message with github_url.
   • If run_passed=False: send failure alert listing failed scenarios and github_url.
   • If consecutive_failures >= 3: prepend 🚨 连续 {n} 轮失败，请人工介入.

---

Circuit Breaker

Track consecutive_failures across runs (persist in your memory or state):

• Increment on run_passed=False.
• Reset to 0 on run_passed=True.
• If consecutive_failures >= 3: send an urgent Feishu alert and pause the cron schedule. Do NOT continue running automatically until a human acknowledges and resets the counter.

---

Feishu Message Templates

Pass:

✅ RobustMQ 故障测试通过
Run ID: {run_id}  时间: {finished_at}
核心场景: 全部通过  总通过率: {pass_rate}%
报告: {github_url}

Fail:

❌ RobustMQ 故障测试失败
Run ID: {run_id}  时间: {finished_at}
失败场景: {failed_scenario_list}
报告: {github_url}

Circuit breaker:

🚨 连续 {consecutive_failures} 轮失败，请人工介入
最后失败: {run_id}  {finished_at}
报告: {github_url}

---

Pitfalls

• Never judge pass/fail on fault-period results (Step 3). Only Step 5 post-recovery validation counts.
• script_format_error ≠ RobustMQ bug. Report it separately; do not inflate the failure count. Fix the script first.
• ROBUSTMQ_HOME must be set before any run. The cluster tool returns an error immediately if it is not — surface it and stop rather than retrying.
• Consecutive failures count whole runs, not individual scenarios. One run with two failed scenarios = 1 failure, not 2.
• eth0 is not universal. The network-delay target interface name varies by host. Verify it before running in a new environment.
• Deploy Key permissions. If report returns push_error, the reports are still written locally at json_path / markdown_path. Investigate the key before declaring the run lost.
• 60-second wait is mandatory. Do not skip or shorten it — RobustMQ leader election and connection re-establishment take time.