test-sh-monitor

test.sh Running and Monitoring Guide

Core Principle

Always run the complete test.sh — never execute individual phases in isolation.

test.sh sets critical environment variables (CARGO_TARGET_DIR, RUSTFLAGS, CONFLUX_BENCH), activates the venv, and manages symlinks. These side effects only take effect during a full run. In particular, CARGO_TARGET_DIR=$ROOT_DIR/build exists only within the test.sh process environment — manually running cargo build inside tools/consensus_bench/ will drop artifacts in the wrong location, causing integration tests to pick up a stale binary.

---

Step 0: Ask for Polling Interval

Before doing anything else, ask the user:

"How often should I check the log? Options: 60s, 120s, or a custom value."

Remember the chosen interval — it will be used in the monitoring loop in Step 3.

---

Step 1: Pre-flight Checks

Confirm these five items before launching to avoid discovering environment problems after a long wait:

# 1. uv is present
command -v uv || echo "MISSING: run: curl -LsSf https://astral.sh/uv/install.sh | sh"

# 2. submodules are initialized (required separately for each new worktree)
git submodule status | grep "^-" && echo "MISSING submodules: run git submodule update --init --recursive" || echo "submodules OK"

# 3. target directory state (must be a symlink, not a real directory)
ls -la target 2>/dev/null || echo "target not present (OK, test.sh will handle)"

# 4. venv (test.sh creates it automatically; this is just a sanity check)
ls .venv 2>/dev/null && echo "venv exists" || echo "venv absent (test.sh will create it)"

# 5. fmt check (fast — catches formatting issues before starting the long build)
bash cargo_fmt.sh -- --check

If fmt check fails, fix formatting before launching test.sh — there is no point waiting for a build that will eventually fail the same check.

If submodules are missing:

git submodule update --init --recursive

---

Step 2: Launch test.sh

Prefer run_in_background: true so the framework owns the process and notifies you on exit, structurally eliminating zombie process issues:

bash -c 'set -o pipefail; bash dev-support/test.sh 2>&1 | tee /tmp/test_run.log'
# (with run_in_background: true)

Why pipe instead of > file 2>&1? test.sh's check functions use tee /dev/stderr internally. When stderr is redirected to a regular file, tee opens /dev/stderr via /proc/self/fd/2, which creates a new file description starting at position 0 — overwriting earlier content (including phase anchors). With a pipe, there is no file position, so tee /dev/stderr appends correctly. pipefail ensures the exit code reflects test.sh, not tee.

Why prefer run_in_background? The framework-owned process is not a child of the current shell, so the framework reaps it — no need to handle zombies in the monitoring loop.

If you must background it manually, do not use kill -0 as the loop condition — it cannot distinguish a running process from a zombie (kill -0 returns 0 for both). Use ps -o stat= instead:

bash -c 'set -o pipefail; bash dev-support/test.sh 2>&1 | tee /tmp/test_run.log' &
PID=$!
while [[ "$(ps -p $PID -o stat= 2>/dev/null)" =~ ^[^Z] ]]; do
    sleep 10
    tail -5 /tmp/test_run.log
done
wait $PID
echo "exit: $?"
tail -50 /tmp/test_run.log

ps -o stat= returns a status character: S/R/D = running normally, Z = zombie, empty = process gone. The regex ^[^Z] exits the loop on either zombie or gone; wait $PID then reaps and returns the true exit code.

---

Step 3: Periodic Monitoring

The core check logic is the same regardless of tool availability:

PASSED=$(grep -c "✓" /tmp/test_run.log 2>/dev/null || echo 0)
FAILED=$(grep -cE "[✖✗]" /tmp/test_run.log 2>/dev/null || echo 0)
RUNNING=$(pgrep -f "bash dev-support/test.sh" > /dev/null 2>&1 && echo "RUNNING" || echo "STOPPED")
PHASE=$(grep -oE "(Phase [0-9]/[0-9]: [^=]+|Fmt check( passed)?)" /tmp/test_run.log 2>/dev/null | tail -1 || echo "unknown")
echo "[$(date '+%H:%M:%S')] $RUNNING phase=[$PHASE] passed=$PASSED failed=$FAILED"
tail -3 /tmp/test_run.log

Keep the check logic simple — count passes/fails, detect phase from anchors, tail the log. Do NOT attempt smart phase inference in the script; leave interpretation to the AI.

Below are two options depending on tool availability. Prefer Option A if the Monitor tool is available — it runs the polling loop as a background task whose stdout lines become chat notifications, eliminating the manual alternating pattern. Option B is the fallback for Claude Code versions that do not have the Monitor tool.

Option A: With Monitor tool

Use Monitor as a log filter. The script runs a polling loop; each echo becomes a notification. Only emit output on noteworthy events — phase changes, new failures, or process exit — to avoid flooding the conversation:

Monitor({
  description: "test.sh watchdog",
  timeout_ms: 3600000,
  persistent: false,
  command: """
PREV_PHASE=""
PREV_FAILED=0
while true; do
  if [ ! -f /tmp/test_run.log ]; then sleep 10; continue; fi
  RUNNING=$(pgrep -f "bash dev-support/test.sh" > /dev/null 2>&1 && echo "yes" || echo "no")
  PASSED=$(grep -c "✓" /tmp/test_run.log 2>/dev/null || echo 0)
  FAILED=$(grep -cE "[✖✗]" /tmp/test_run.log 2>/dev/null || echo 0)
  PHASE=$(grep -oE "(Phase [0-9]/[0-9]: [^=]+|Fmt check( passed)?)" /tmp/test_run.log 2>/dev/null | tail -1)

  if [ "$PHASE" != "$PREV_PHASE" ]; then
    echo "[$(date '+%H:%M:%S')] $PHASE | passed=$PASSED failed=$FAILED"
    PREV_PHASE="$PHASE"
  fi
  if [ "$FAILED" -gt "$PREV_FAILED" ]; then
    echo "[$(date '+%H:%M:%S')] NEW FAILURE: failed=$FAILED (was $PREV_FAILED)"
    grep -E "[✖✗]" /tmp/test_run.log 2>/dev/null | tail -5
    PREV_FAILED=$FAILED
  fi
  if [ "$RUNNING" = "no" ]; then
    echo "[$(date '+%H:%M:%S')] EXITED | passed=$PASSED failed=$FAILED"
    tail -10 /tmp/test_run.log
    exit 0
  fi
  sleep <interval>
done
"""
})

Option B: Without Monitor tool

Use the foreground Bash + background sleep alternating pattern:

Step A — foreground log check (no sleep):

PASSED=$(grep -c "✓" /tmp/test_run.log 2>/dev/null || echo 0)
FAILED=$(grep -cE "[✖✗]" /tmp/test_run.log 2>/dev/null || echo 0)
RUNNING=$(pgrep -f "bash dev-support/test.sh" > /dev/null 2>&1 && echo "RUNNING" || echo "STOPPED")
PHASE=$(grep -oE "(Phase [0-9]/[0-9]: [^=]+|Fmt check( passed)?)" /tmp/test_run.log 2>/dev/null | tail -1 || echo "unknown")
echo "[$(date '+%H:%M:%S')] $RUNNING phase=[$PHASE] passed=$PASSED failed=$FAILED"
tail -3 /tmp/test_run.log

Step B — background timer (run_in_background: true):

sleep <interval chosen in Step 0>

On timer notification → run Step A → run Step B → repeat.

Why this pattern? The foreground Bash call returns immediately without blocking the conversation; every report is visible in real time. The background sleep is purely a timer. Common mistakes: background bash loop (intermediate output invisible), foreground sleep (blocks the conversation), background Agent for polling (same problem as background bash loop).

---

Phases

Pre-check: Fmt

• Anchor: === Fmt check ===
• Success anchor: === Fmt check passed ===
• Behavior: runs cargo_fmt.sh -- --check (nightly rustfmt across all workspaces). Fast (seconds). On failure, process exits immediately — no success anchor, no build phases.
• Note: Step 1 pre-flight already runs this check independently. If you see fmt failure here, it means the pre-flight was skipped or code changed between pre-flight and launch.

Phase 1: cargo build (main project)

• Anchor: === Phase 1/4: Building main project ===
• Log signal: continuous Compiling xxx (from cargo stderr; absent when build is cached)
• Success anchor: === Phase 1/4: Build succeeded ===
• Failure: error[E...] from cargo stderr — process exits immediately (no success anchor)

Phase 2: cargo build consensus_bench

• Anchor: === Phase 2/4: Building consensus_bench ===
• Success anchor: === Phase 2/4: Build succeeded ===

Phase 3: test_all.py (integration tests) — requires active monitoring

• Anchor: === Phase 3/4: Integration tests ===
• Key behavior: parallel scheduling; a single test failure does not exit the process. Early break triggers: (1) >5 failures in a single round, or (2) any test that fails twice across retry rounds
• Process alive ≠ tests passing — tail the log regularly and count ✖
• Success anchor: === Phase 3/4: Integration tests passed ===
• Failure: any ✖; exit code 1 or 80

Phase 4: pytest

• Anchor: === Phase 4/4: Pytest ===
• Watch for: FAILED and ERROR (pytest uses these, not ✖)
• Success anchor: === Phase 4/4: Pytest passed ===

---

Log Keyword Reference

Not in the log: Build succeeded. / Build failed. / Integration test failed. / Pytest failed. — these strings are captured into shell variables by $() and never printed to stdout/stderr. Use the === Phase === anchors instead.

---

New Worktree Quick Start

When the current branch is already checked out by the main worktree, use --detach to avoid the branch conflict:

# 1. Create worktree
git worktree add --detach /home/ubuntu/worktrees/my-test HEAD

# 2. Initialize submodules (required for every new worktree)
cd /home/ubuntu/worktrees/my-test
git submodule update --init --recursive

# 3. (Optional) Reuse build artifacts to skip 30–50 min of compilation
#    Requires both worktrees to be on the same commit
ln -s /conflux-rust/build /home/ubuntu/worktrees/my-test/build

# 4. Launch (run_in_background: true)
bash -c 'set -o pipefail; bash dev-support/test.sh 2>&1 | tee /tmp/test_run.log'

---

Three Monitoring Layers — All Required

1. Process layer: is the process still alive? (compile failures exit quickly)
2. Log layer: tail regularly and interpret keywords by phase
3. Semantic layer: in Phase 3, "process alive" does not mean "tests passing" — check the ✖ count