zeus-debugging-playbook

Zeus Debugging Playbook

Symptom → triage for Zeus's recurring failure modes. Each row names the first thing to check, the likely cause, the fix pattern, and (where one exists) the commit that proves the story. Deep incident narratives live in zeus-failure-archaeology — this skill gives you the fast path.

When to use / When NOT to use

Use this skill when something is broken and you need to find out why: a runtime misbehavior, a CI failure you can't reproduce locally, a crash, a hang, a silently-no-op feature.

Do NOT use this skill for:

Jargon in one line each

• Backend / implementation: Zeus talks to 7 kinds of Lightning nodes (embedded-lnd, ldk-node, lnd, lightning-node-connect, cln-rest, lndhub, nostr-wallet-connect). The active one is the string settingsStore.implementation.
• BackendUtils dispatch: utils/BackendUtils.ts routes every node call to the active backend class in backends/. If the class lacks the method, call() returns false synchronously — it does not throw.
• supports*() flags: per-backend capability methods (e.g. supportsMessageSigning()); views must gate features on these, never on the implementation string.
• MobX store: an observable state singleton in stores/ (e.g. SettingsStore); views react to its fields.
• Keychain: iOS/Android secure credential storage; Zeus persists nearly everything there via storage/index.ts, including the whole settings blob.
• Hermes: the JavaScript engine React Native uses in Zeus; it has a shallow native stack, so heavy regex backtracking overflows it.
• Fabric / New Architecture: React Native's rewritten rendering layer, ON in Zeus since RN 0.83.1 (commit d13ebc2cd); it is stricter about Animated/layout usage than the old renderer.
• Tor: anonymity network; Zeus can route node REST calls through it (enableTor per node).
• Embedded LND: a full LND Lightning node compiled to a mobile library (gomobile) running inside the app; LDK Node is the same idea built on the Lightning Dev Kit (Rust, uniffi FFI).
• Macaroon: LND's bearer auth token, sent as a header on REST calls.
• gRPC streaming: long-lived request channels used by the embedded-LND native layer (Blixt-derived code in android/app/src/main/java/com/zeus/ and ios/LndMobile/).

Before you debug — 4 questions

1. Which backend? Most "bugs" are missing capability gates. Reproduce on (or at least reason about) the specific implementation. A feature working on lnd and dead on lndhub is a gating issue, not a logic issue.
2. Which platform? The embedded-LND native layer is separate code per platform (LndMobileService.java vs ios/LndMobile/Lnd.swift); NWC background services, stealth mode, and keychain-iCloud behavior are platform-specific.
3. Fresh install or upgrade? One-shot migrations (the MOD_KEY blocks in utils/MigrationUtils.ts) only run once per install. Upgrade-only bugs live there — a fresh install will never reproduce them. See zeus-storage-and-migrations.
4. Tor on or off? Tor changes the request path (dedup-cache retention, TLS rules, WebSocket bypass). Toggle it as a discriminating experiment.

Also remember: stores/, views/, components/, and backends/ have zero test coverage (tests exist only for utils/, models/, lndmobile/) — a green yarn test proves nothing about a store bug. See zeus-validation-and-qa.

Triage table

Discriminating experiments (rows that need more than one line)

Rows 1–2: missing backend method vs missing gate

BackendUtils.call() in utils/BackendUtils.ts returns false synchronously for a method the active backend lacks — the dispatcher code and full consequence catalog are owned by zeus-backends-and-capabilities §1. Discriminate in three steps:

# 1. Is the method on the active backend class? (LND is the implicit base;
#    EmbeddedLND and LndHub extend it — check the parent too)
grep -n 'theMethod' backends/LndHub.ts backends/LND.ts

# 2. Is there a dispatch wrapper at all? (a typo'd name silently no-ops)
grep -n 'theMethod' utils/BackendUtils.ts

# 3. At runtime: log what came back
console.log(typeof result, result); // "boolean false" => missing method

Known dead dispatch as of 2026-07-06: payLightningInvoiceStreaming is declared in utils/BackendUtils.ts but implemented by no backend — it always returns false. Inheritance leaks (un-overridden LND REST methods firing on EmbeddedLND) are in zeus-backends-and-capabilities.

Row 3: shallow-merge settings loss

stores/SettingsStore.ts updateSettings does { ...existingSettings, ...newSetting } — top level only. updateSettings({ privacy: { lurkerMode: true } }) erases every other privacy field for the user. Discriminating experiment: diff the settings blob before/after the call (see zeus-diagnostics-and-tooling for inspecting it). If a nested group shrank, you found it. Repair for already-shipped damage requires a migration — zeus-storage-and-migrations, and note storage changes are maintainer-gated (zeus-change-control).

Row 4: stuck connecting

views/Wallet/Wallet.tsx fetchData() opens with:

if (fetchLock) return;
SettingsStore.fetchLock = true;

and SettingsStore.connecting initializes true. getSettingsAndNavigate() runs on Wallet-screen focus only for initial load, POS exit, or an explicit settings refresh (this.state.initialLoad || SettingsStore.posWasEnabled || SettingsStore.triggerSettingsRefresh), and is skipped while the in-flight _navigating guard flag is set. If any code path between lock-set and lock-release throws unhandled or early-returns, the app hangs on the loading state forever — and because focus events won't re-trigger the call outside those three conditions, nothing retries it. SettingsStore.setConnectingStatus(true) clears fetchLock and the request cache — that's why "go to settings and back" sometimes unsticks it (a diagnostic signal, not a fix). Trap story: the UNSAFE-lifecycle refactor fdad118ed caused exactly this; 93227029e fixed it three weeks later. Deep narrative: zeus-failure-archaeology.

Rows 5–7: CI failures that lie about their cause

Green PR = 4 checks (Test, Lint, Prettier, Typescript Check) = yarn verify. Anatomy and jest traps are owned by zeus-validation-and-qa; the three that masquerade as something else:

• Lint runs a jest test: check-styles.test.ts (repo root) is excluded from yarn test via testPathIgnorePatterns and re-included by yarn lint with --testPathIgnorePatterns=. Failure text lists the offending .tsx files.
• Prettier/Lint double failure: prettier violations also fail ESLint (prettier/prettier is an error rule in eslint.config.js). Fix formatting once, both clear.
• Test on a new dep: extend the transformIgnorePatterns whitelist in package.json; current members include react-native, @react-native, react-native-blob-util, nostr-tools, @noble, @scure, uuid, etc.

Row 10: Tor request dedup cache retains rejections

Both backends/LND.ts and backends/CLNRest.ts keep a module-level const calls = new Map<string, Promise<any>>() keyed by url + JSON.stringify(body). The non-Tor branch deletes the entry on both success and failure (a .catch on the raced promise). The Tor branch deletes only in .then:

doTorRequest(...).then((response: any) => {
    calls.delete(id);   // no .catch — a rejection stays cached
    return response;
})

Discriminating experiment: same request, Tor off → retry gets a fresh attempt; Tor on → retry replays the identical rejection instantly (suspiciously fast failure = cached). Cache is cleared on reconnect (SettingsStore.setConnectingStatus(true) → BackendUtils.clearCachedCalls()).

Row 13: embedded LND startup — and getting real logs

lndmobile/index.ts readLndLog is a stub (return [''];, marked TODO remove) — do not build diagnostics on it. The actual log path is what views/Settings/EmbeddedNode/LNDLogs.tsx does: NativeModules.LndMobileTools.tailLog(...) plus the 'lndlog' event listener. In-app: the embedded node settings expose the LND Logs screen. Full log-capture recipes: zeus-diagnostics-and-tooling. The DB backend is chosen per node (isSqlite → db.backend=sqlite|bolt written into lnd.conf by utils/LndMobileUtils.ts); the SQLite/Bolt history (ef4a42112 → 3c6f4175e → d475ef2b4 → 3e04e6bf2) is chronicled in zeus-failure-archaeology.

Row 15: stream-write success is unverifiable

android/app/src/main/java/com/zeus/LndMobileService.java (line 279 as of 2026-07-06):

try {
  s.send(payload);
} catch (Throwable error) {
  // TODO(hsjoberg): Handle errors
}
// ...then unconditionally reports MSG_GRPC_STREAM_WRITE_RESULT

ios/LndMobile/Lnd.swift (line 334): try write?.send(bytes) // TODO(hsjoberg): Figure out whether send returns a BOOL? — thrown errors are forwarded, but a false-returning send would still call back success. Verification pattern: after any stream write you care about, confirm the effect with an independent read RPC.

Grep hygiene

When searching the repo for debt markers, symbols, or error strings, exclude generated and vendored code or you'll drown in noise:

grep -rn "TODO" . \
  --include='*.ts' --include='*.tsx' \
  --exclude-dir=node_modules --exclude-dir=zeus_modules \
  --exclude-dir=proto --exclude-dir=android --exclude-dir=ios

• ios/LdkNodeMobile/LDKNode.swift and android/app/src/main/java/org/lightningdevkit/ldknode/ldk_node.kt are uniffi-generated bindings (18 TODO hits each as of 2026-07-06) — never "fix" them by hand.
• proto/lightning.js is generated protobuf output.
• zeus_modules/ is vendored (lnc-rn has its own node_modules inside it).
• utils/AddressUtils.test.ts contains XXX inside a base64 PSBT fixture (~line 923) — a false positive for placeholder hunts.
• fetch-libraries.sh defines a shell function literally named jq() (it's python3) — misleading grep target.

Provenance and maintenance

Facts verified 2026-07-06 against master c5fd094fb (v13.1.3-alpha) by reading the cited files and running the commands below. All cited commit hashes verified as ancestors of that commit via git merge-base --is-ancestor <hash> HEAD.

Re-verification one-liners for volatile facts: