OpenParable
Skip to content
← Back to skills

Edenfs Systemd Triage

DevOps & Security
edenfs-systemd-triage

Edenfs Systemd Triage is an Agent Skill for devops & security published in the facebook/sapling repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Triage systemd-managed EdenFS issues on Linux devservers and OnDemands. Use when investigating EdenFS service failures, unexpected restarts, systemctl errors, edenfs_upgrade/edenfs_restarter problems, or when a user reports EdenFS is down on a systemd-enabled host. Also use when someone asks how systemd-managed EdenFS works, how to monitor it, or how to check its health. Use when asked to build a timeline of EdenFS lifecycle events, show edenfs restart history, or understand how edenfs reached its current state. Trigger on mentions of edenfs systemd, edenfs@ service, edenfs_upgrade timer, edenfs auto-restart, eden status --debug, systemctl edenfs, edenfs lifecycle management, edenfs timeline, edenfs restart history, or "what happened to edenfs".

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/facebook/sapling/blob/HEAD/eden/.llms/skills/edenfs-systemd-triage/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/edenfs-systemd-triage, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

EdenFS Systemd Triage

This skill helps EdenFS team members understand, monitor, and triage systemd-managed EdenFS.

Quick Orientation

Systemd manages the EdenFS daemon lifecycle on Linux (devservers and OnDemands). When EdenFS exits unexpectedly (crash, OOM-kill), systemd automatically restarts it — no user intervention needed.

Two systemd components to know:

ComponentWhat it doesScope
[email protected]User-scoped service managing the edenfs daemon lifecycle (auto-restart on failure)Per-user (systemctl --user)
edenfs_upgrade.timerSystem-scoped hourly timer that runs edenfs_restarter to upgrade edenfs gracefullySystem-wide (systemctl)

Config gate: [experimental] systemd-managed-lifecycle = true in eden config.

How to Use This Skill

Read the reference file that matches your need:

NeedReference FileWhen to Read
Understand the architecturereferences/architecture.mdHow systemd-managed EdenFS works, service unit file, lifecycle operations
Check health & monitorreferences/monitoring.mdScuba queries, dashboards, success rate metrics, rollout monitoring
Build a lifecycle timelinereferences/timeline.mdReconstruct chronological EdenFS events from Scuba + local logs + systemd properties to understand how the system reached its current state
Triage a specific failurereferences/triage-playbook.mdStep-by-step procedures for common failure scenarios
Identify known failure patternsreferences/common-failures.mdError signatures, root causes, and fixes for known issues

First-Response Checklist

When triaging an EdenFS systemd issue, start here:

  1. Is the user on systemd? Check: eden config | grep systemd-managed
  2. What's the service status? Check: eden status --debug
  3. What does Scuba say? Check systemctl action failures: https://fburl.com/scuba/edenfs_events/4tf01a9c
  4. What do the logs say? Check: /var/facebook/logs/edenfs_upgrade.log and /var/log/messages

If you need to run commands on a user's machine via sush, you cannot use su — you must use:

machinectl shell <username>@.host /usr/local/bin/eden status --debug

Diagnostic Commands (Safe to Run Locally)

These are read-only and safe to execute automatically:

# Check if systemd-managed
eden config | grep systemd-managed

# Full service status with systemd details
eden status --debug

# Check eden version mismatch
eden version

# Check systemd service properties (restart policy, crash counters, timestamps)
systemctl --user show edenfs@home-$(whoami)-local-.eden.service \
  --property=Id,Type,Restart,RestartUSec,StartLimitIntervalUSec,StartLimitBurst,NRestarts,ExecMainStartTimestamp,ExecMainPID,ExecMainCode,ExecMainStatus,ActiveState,SubState,Result,InvocationID,ActiveEnterTimestamp,ActiveExitTimestamp,InactiveEnterTimestamp,InactiveExitTimestamp

# Check eden logs for recent errors
eden debug log | tail -50

# Check startup log
cat <state_dir>/.edenfs_startup.log

# Check system messages for edenfs service events
grep 'edenfs@' /var/log/messages | tail -20

# Check edenfs_upgrade logs
tail -50 /var/facebook/logs/edenfs_upgrade.log

# Check kernel OOM kills
dmesg | grep -i -E '(edenfs|oom|killed)' | tail -20

# Check dbus connectivity (needed for systemctl --user)
python3 -c "import socket; s=socket.socket(socket.AF_UNIX,socket.SOCK_STREAM); s.settimeout(1); s.connect('/run/user/$(id -u)/bus'); print('alive'); s.close()"

# Check linger (needed for user services to persist)
loginctl show-user $(whoami) --property=Linger

Hostname for Scuba Queries

CRITICAL: Scuba uses the short hostname (e.g., devvm21611.cco0), NOT the FQDN returned by hostname (e.g., devvm21611.cco0.facebook.com). Always strip the .facebook.com suffix:

hostname | sed 's/\.facebook\.com$//'

Querying Scuba with the FQDN will silently return zero results.