kurtosis-devnet

Kurtosis Devnet

Run Ethereum consensus/execution client devnets via Kurtosis + ethereum-package.

Prerequisites

• Kurtosis CLI installed
• Docker running with sufficient resources (8GB+ RAM recommended for multi-client devnets)

Quick Start

# Start devnet from config
# Always use --image-download always to ensure external images are up to date
kurtosis run github.com/ethpandaops/ethereum-package \
  --enclave <name> \
  --args-file network_params.yaml \
  --image-download always

# List enclaves
kurtosis enclave ls

# Inspect services
kurtosis enclave inspect <name>

# View logs
kurtosis service logs <enclave> <service-name>
kurtosis service logs <enclave> <service-name> --follow

# Cleanup
kurtosis enclave rm -f <name>
# Or clean all enclaves
kurtosis clean -a

Config File (network_params.yaml)

See references/config-reference.md for the full config structure.

Key sections:

• participants: list of CL+EL client pairs with images, flags, validator counts
• network_params: fork epochs, slot time, network-level settings
• additional_services: dora (explorer), assertoor (testing), prometheus, grafana
• assertoor_params: automated chain health checks
• port_publisher: expose CL/EL ports to host

Building Custom Client Images

When testing local Lodestar branches, build a Docker image first:

# Fast build (recommended for iteration)
cd ~/lodestar && docker build -t lodestar:custom -f Dockerfile.dev .

# Production build (slower, for final validation only)
cd ~/lodestar && docker build -t lodestar:custom .

# Then reference in config:
# cl_image: lodestar:custom

Always use Dockerfile.dev for iterative development. It caches dependency layers and rebuilds in seconds vs minutes for the production Dockerfile. Only use the production Dockerfile for final validation or debugging build issues.

Service Naming Convention

Kurtosis names services as: {role}-{index}-{cl_type}-{el_type}

Examples:

• cl-1-lodestar-reth — first CL node (Lodestar with Reth EL)
• el-1-reth-lodestar — corresponding EL node
• vc-1-lodestar-reth — validator client

Accessing Services

After kurtosis enclave inspect <name>, find mapped ports:

# CL beacon API (find actual port from inspect output)
curl http://127.0.0.1:<mapped-port>/eth/v1/node/syncing

# Or use port_publisher for predictable ports:
# port_publisher:
#   cl:
#     enabled: true
#     public_port_start: 33000  # cl-1=33000, cl-2=33005, etc.
#   el:
#     enabled: true
#     public_port_start: 32000

Port publisher assigns sequential ports (step of 5 per service).

Assertoor (Automated Testing)

Add to config:

additional_services:
  - assertoor

assertoor_params:
  run_stability_check: true # chain stability, finality, no reorgs
  run_block_proposal_check: true # every client pair proposes a block

Check results via the assertoor web UI (port shown in kurtosis enclave inspect).

Common Devnet Patterns

Fork Transition Testing

network_params:
  electra_fork_epoch: 0
  fulu_fork_epoch: 1 # fork at epoch 1 (slot 32)
  seconds_per_slot: 6 # faster for testing

Mixed-Client Topology (Cross-Client Interop)

participants:
  - cl_type: lodestar
    el_type: reth
    count: 2
    validator_count: 128
  - cl_type: lighthouse
    el_type: geth
    count: 2
    validator_count: 128

Observer Nodes (No Validators)

- cl_type: lodestar
  cl_image: lodestar:custom
  el_type: reth
  count: 1
  validator_count: 0 # observer-only

Supernode Mode

Set supernode: true to run beacon+validator in a single process (faster startup, simpler topology):

- cl_type: lodestar
  el_type: reth
  supernode: true
  validator_count: 128

Extra CL/VC Params

cl_extra_params:
  - --targetPeers=8
  - --logLevel=debug
vc_extra_params:
  - --suggestedFeeRecipient=0x...

Monitoring & Debugging

# Stream logs from a specific service
kurtosis service logs <enclave> cl-1-lodestar-reth --follow

# Save all CL logs for analysis
for svc in $(kurtosis enclave inspect <enclave> 2>/dev/null | grep -oE 'cl-[0-9]+-[^[:space:]]+'); do
  kurtosis service logs <enclave> $svc > "/tmp/${svc}.log" 2>&1
done

# Dora explorer (if enabled) — find port via inspect output

# Check chain finality
curl -s http://127.0.0.1:<port>/eth/v1/beacon/states/head/finality_checkpoints | jq

# Check peer count
curl -s http://127.0.0.1:<port>/eth/v1/node/peers | jq '.data | length'

# Check sync status
curl -s http://127.0.0.1:<port>/eth/v1/node/syncing | jq

Wait for Finality

Finality typically takes 2-3 epochs after genesis. With seconds_per_slot: 6 and 32 slots/epoch:

• 1 epoch ≈ 192s (3.2 min)
• First finalization ≈ epoch 3-4 boundary (≈10-13 min)

Monitor:

curl -s http://<port>/eth/v1/beacon/states/head/finality_checkpoints | jq '.data.finalized.epoch'

Acceptance Criteria Pattern

For interop validation, define acceptance criteria before running the soak test:

# Example: check for zero errors across all CL nodes
for svc in cl-1-lodestar-reth cl-2-lodestar-reth; do
  echo "=== $svc ==="
  kurtosis service logs <enclave> $svc 2>&1 | grep -c "ERROR" || echo "0 errors"
done

Troubleshooting

Tips

• Start small: 2+2 nodes for iteration, scale up for final validation
• Test both roles: Validator nodes show different bugs than observers — always check both
• Multiple soak passes: First clean run may miss edge cases. Run extended soaks (hours).
• kurtosis clean -a between runs — never use docker system prune (it can remove Kurtosis engine images)
• Debug logs: Set global_log_level: "debug" for verbose output from all nodes
• Per-participant log levels: Use cl_log_level / el_log_level on individual entries for targeted debugging

---

Self-Maintenance

If any commands, file paths, URLs, or configurations in this skill are outdated or no longer work, update this SKILL.md with the correct information after completing your current task. Skills should stay accurate and self-healing — fix what you find broken.