# Hive Test

> Hive Test is an Agent Skill for testing & quality published in the erigontech/erigon repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Run Ethereum Hive integration tests against a local Erigon build, including engine, RPC compatibility, standard EEST, BAL/Amsterdam EEST, and RLP suites. Use when setting up an ephemeral Hive environment, selecting or running Hive suites, interpreting Hive failures, or cleaning up Hive test resources.

- Skill name: `hive-test`
- Category: [Testing & Quality](https://www.openparable.com/skill/testing-quality/1)
- Source: https://github.com/erigontech/erigon/blob/HEAD/.claude/skills/hive-test/SKILL.md
- Repository: erigontech/erigon (3,582 GitHub stars when collected)

## SKILL.md

# Skill: hive-test

Run Ethereum Hive integration tests against a local Erigon build. Works from a clean
environment -- no pre-existing hive installation required.

## Metadata
- user-invocable: true
- description: Run Hive integration tests (engine, rpc, eest) against local Erigon
- allowed-tools: Bash, Read, Write, Edit, Glob, Grep

## Overview

This skill sets up an ephemeral Hive test environment, builds a local Erigon Docker
image, runs the requested test suites, reports results, and cleans up containers.

## Arguments

The user may specify one or more test suites in any combination:

### Individual suites
| Suite name | Simulator | Description |
|-----------|-----------|-------------|
| `exchange-capabilities` | ethereum/engine | Engine exchange-capabilities |
| `withdrawals` | ethereum/engine | Engine withdrawals |
| `cancun` | ethereum/engine | Engine cancun |
| `api` | ethereum/engine | Engine API |
| `auth` | ethereum/engine | Engine auth |
| `rpc-compat` | ethereum/rpc | RPC compatibility |
| `eest` | ethereum/eels/consume-engine | Execution Spec Tests (version auto-discovered) |
| `eest-devnet` | ethereum/eels/consume-engine | EEST devnet (BAL/glamsterdam) fixtures (version auto-discovered) |
| `eest-rlp` | ethereum/eels/consume-rlp | EEST RLP block import (BlockchainTest, all forks) |

### Groups
| Group name | Expands to |
|-----------|------------|
| `engine` | exchange-capabilities, withdrawals, cancun, api, auth |
| `all` | Every suite listed above |

### Examples
- `/hive-test api` - Run just the engine API suite
- `/hive-test withdrawals api` - Run withdrawals and API suites
- `/hive-test engine` - Run all engine suites
- `/hive-test engine rpc-compat` - Run all engine suites plus rpc-compat
- `/hive-test eest-devnet` - Run devnet EEST tests (BAL/glamsterdam)
- `/hive-test eest-rlp` - Run EEST RLP block-import tests
- `/hive-test all` - Run everything

### Options
- **erigon-path** - Path to local erigon source (default: current working directory)
- **branch=BRANCH** - Clone erigon from a remote branch instead of using the local
  working directory. The branch is cloned from `https://github.com/erigontech/erigon.git`
  into the hive client directory. Example: `/hive-test api branch=fix/my-feature`
- **eest-version=VERSION** - Pin EEST fixtures version (e.g. `v5.3.0`). Default: auto-discover latest.
- **bal-version=VERSION** - Pin BAL fixtures version (e.g. `bal@v5.1.0`). Default: auto-discover latest.

## Expected Failures (CI thresholds)

Sources of truth: `.github/workflows/test-hive.yml` (`max-allowed-failures` per matrix
entry) for engine + rpc-compat suites, `.github/workflows/test-hive-eest.yml`
(`max-failures` per matrix entry, currently 0 everywhere) for eest shards.

| Suite | Max Allowed Failures |
|-------|---------------------|
| exchange-capabilities | 0 |
| withdrawals | 0 |
| cancun | 3 (2 known Hive/Geth secondary-client failures + 1 known parallelism flake) |
| api | 0 |
| auth | 0 |
| rpc-compat | 0 |
| eest (consume-engine) | 0 |
| eest-rlp | 0 |
| eest-devnet (CI shard: `glamsterdam-devnet`) | 0 |

Note: Failure counts are version-dependent and may change with newer fixtures.
The CI `glamsterdam-devnet` shard runs BAL EIPs (`7708|7778|7843|7928|7954|7976|7981|8024|8037`)
against the URL and hive `branch` pinned under the `eest_devnet`
entry in `test-fixtures.json` (currently `tests-bal@v7.3.2` / `devnets/bal/7`),
with `--experimental.bal` enabled on the erigon side. Reproduce locally by
aligning the invocation with those values — `make eest-devnet` reads them
from the manifest via `jq` and applies them automatically.

## Procedure

### Phase 0: Discover Versions

Before setup, discover the latest EEST fixture versions from GitHub. Skip this phase
if the user provided explicit version overrides (`eest-version=`, `bal-version=`).

```bash
# Latest standard EEST fixtures (tag matching v*.*.*)
EEST_VERSION=$(curl -s https://api.github.com/repos/ethereum/execution-spec-tests/releases \
  | jq -r '[.[] | select(.tag_name | test("^v[0-9]+\\.[0-9]+\\.[0-9]+$"))][0].tag_name')

# Latest BAL fixtures (tag matching bal@v*.*.*)
BAL_TAG=$(curl -s https://api.github.com/repos/ethereum/execution-spec-tests/releases \
  | jq -r '[.[] | select(.tag_name | startswith("bal@"))][0].tag_name')
BAL_BRANCH="tests-${BAL_TAG}"
BAL_TAG_URLENC=$(echo "$BAL_TAG" | sed 's/@/%40/')
BAL_FIXTURES_URL="https://github.com/ethereum/execution-spec-tests/releases/download/${BAL_TAG_URLENC}/fixtures_bal.tar.gz"
```

Then check whether the EEST mapper at the discovered tag already has the exception
entries Erigon needs. If not, the `disable_strict_exception_matching` workaround is
required:

```bash
MAPPER_URL="https://raw.githubusercontent.com/ethereum/execution-specs/refs/tags/${BAL_BRANCH}/packages/testing/src/execution_testing/client_clis/clis/erigon.py"
if curl -sf "$MAPPER_URL" | grep -q "GAS_USED_OVERFLOW"; then
    DISABLE_STRICT=""
    echo "Mapper has BAL exception entries — strict matching enabled"
else
    DISABLE_STRICT='--sim.buildarg disable_strict_exception_matching=erigon'
    echo "Mapper missing BAL exception entries — strict matching disabled for erigon"
fi
```

Log the discovered versions:
```
EEST: $EEST_VERSION | BAL: $BAL_TAG (branch: $BAL_BRANCH) | Strict matching: enabled/disabled
```

### Phase 1: Setup

1. **Determine erigon source path.** Default is the current git working directory.
   Verify it contains a `Makefile` and `go.mod` with `erigontech/erigon`.

2. **Choose a work directory.** Use `mktemp -d /tmp/hive-test-XXXXXX` for isolation.

3. **Clone hive:**
   ```bash
   WORKDIR=$(mktemp -d /tmp/hive-test-XXXXXX)
   cd "$WORKDIR"
   git clone --depth 1 https://github.com/ethereum/hive.git
   cd hive
   ```

4. **Copy or clone the erigon source into hive:**

   If `branch=BRANCH` was specified, clone that branch:
   ```bash
   git clone --depth 1 --branch "$BRANCH" \
     https://github.com/erigontech/erigon.git clients/erigon/erigon
   ```

   Otherwise, copy the local source:
   ```bash
   # Use rsync to copy, excluding build artifacts and .git
   rsync -a --exclude='.git' --exclude='build/' --exclude='temp/' \
     "$ERIGON_PATH/" clients/erigon/erigon/
   ```

5. **Install Dockerfile.local** for local builds:
   Ensure `clients/erigon/Dockerfile.local` exists with the correct content.
   Key requirements:
   - Base image: `golang:1.25.7-trixie` (Debian, not Alpine)
   - Build command: `make erigon`
   - Runtime: `debian:13-slim` with `bash curl jq libstdc++6 libgcc-s1`

   If `clients/erigon/Dockerfile.local` doesn't already exist, write the correct version:
   ```dockerfile
   FROM golang:1.25.7-trixie AS builder
   ARG local_path=erigon
   COPY $local_path erigon
   RUN apt-get update && apt-get install -y bash build-essential ca-certificates git \
       && cd erigon && make erigon \
       && cp build/bin/erigon /usr/local/bin/erigon

   FROM debian:13-slim
   COPY --from=builder /usr/local/bin/erigon /usr/local/bin/
   RUN apt-get update && apt-get install -y bash curl jq libstdc++6 libgcc-s1 && rm -rf /var/lib/apt/lists/*
   RUN erigon --version | sed -e 's/erigon version \(.*\)/\1/' > /version.txt
   COPY genesis.json /genesis.json
   COPY mapper.jq /mapper.jq
   COPY erigon.sh /erigon.sh
   COPY enode.sh /hive-bin/enode.sh
   RUN chmod +x /erigon.sh /hive-bin/enode.sh
   EXPOSE 8545 8546 8551 30303 30303/udp
   ENTRYPOINT ["/erigon.sh"]
   ```

6. **P2P protocol configuration:**
   Do NOT add `--p2p.protocol` flags to erigon.sh. Let erigon use its default
   protocol negotiation.

6b. **Parallel execution for Amsterdam (BAL) blocks:**
   Verify that `erigon.sh` enables parallel execution when Amsterdam is configured.
   BAL validation only runs in the parallel execution path. If the following block
   is missing from `erigon.sh`, add it after the `--sync.parallel-state-flushing` line:
   ```bash
   # Enable parallel execution for Amsterdam (BAL) blocks.
   # BAL validation only runs in the parallel execution path.
   if [ "$HIVE_AMSTERDAM_TIMESTAMP" != "" ]; then
       export ERIGON_EXEC3_PARALLEL=true
   fi
   ```
   Without this, Amsterdam blocks run through serial execution which has no BAL
   validation, causing `test_bal_invalid` tests to incorrectly return VALID.

7. **Create client config file:**
   ```bash
   cat > erigon-local.yaml <<'EOF'
   - client: erigon
     dockerfile: local
   EOF
   ```

8. **Build hive binary:**
   ```bash
   go build .
   ```

### Phase 2: Run Tests

Always use `--sim.parallelism 12` for all suites to maximize throughput.

When running multiple suites, launch **separate hive sessions in parallel** (as
background shell commands) whenever the suites use different simulators. This gives
`runs × parallelism` total concurrency. Suites using the same simulator can be combined
with `--sim.limit "suite1|suite2|..."`.

**Engine suites** (sim: `ethereum/engine`) — combine all with `|`:
```bash
./hive --client-file erigon-local.yaml --sim ethereum/engine \
  --sim.limit "exchange-capabilities|withdrawals|cancun|api|auth" \
  --sim.parallelism 12 --sim.timelimit 30m
```

**RPC compat** (sim: `ethereum/rpc`, limit: `compat`):
```bash
./hive --client-file erigon-local.yaml --sim ethereum/rpc \
  --sim.limit "compat" --sim.parallelism 12 --sim.timelimit 30m
```

**EEST** (sim: `ethereum/eels/consume-engine`):
```bash
# $EEST_VERSION discovered in Phase 0 (e.g. v5.4.0), or user-provided via eest-version=
./hive --client-file erigon-local.yaml \
  --sim ethereum/eels/consume-engine \
  --sim.parallelism=12 --docker.nocache=true \
  --sim.buildarg fixtures=https://github.com/ethereum/execution-spec-tests/releases/download/${EEST_VERSION}/fixtures_develop.tar.gz \
  --sim.timelimit 60m
```

**EEST BAL** (sim: `ethereum/eels/consume-engine`, amsterdam filter):
```bash
# $BAL_BRANCH, $BAL_FIXTURES_URL, $DISABLE_STRICT discovered in Phase 0
# (e.g. BAL_BRANCH=tests-bal@v5.2.0), or user-provided via bal-version=
./hive --client-file erigon-local.yaml \
  --sim ethereum/eels/consume-engine \
  --sim.limit=".*amsterdam.*" \
  --sim.parallelism=12 --docker.nocache=true \
  --sim.buildarg branch=${BAL_BRANCH} \
  --sim.buildarg fixtures=${BAL_FIXTURES_URL} \
  ${DISABLE_STRICT} \
  --sim.timelimit 60m
```

**EEST RLP** (sim: `ethereum/eels/consume-rlp`):

Tests block import via RLP-encoded blocks loaded at client startup (the historical
sync code path). Uses the `BlockchainTest` fixture format and covers all forks
including pre-merge, complementary to consume-engine which only covers Paris+.
See https://eest.ethereum.org/main/running_tests/running/#engine-vs-rlp-simulator.

The full RLP test set is too large to run end-to-end in CI — always pass a
`--sim.limit` regex narrowing the scope. CI mirrors this by running only
`.*eip2930_access_list.*`. For local debugging, target a single EIP / opcode
group similarly. Fixtures come from the same `fixtures_develop.tar.gz` archive
as consume-engine.

```bash
# $EEST_VERSION discovered in Phase 0 (e.g. v5.4.0), or user-provided via eest-version=
# Replace the sim.limit regex to scope to the area under test.
./hive --client-file erigon-local.yaml \
  --sim ethereum/eels/consume-rlp \
  --sim.limit=".*eip2930_access_list.*" \
  --sim.parallelism=12 --docker.nocache=true \
  --sim.buildarg fixtures=https://github.com/ethereum/execution-spec-tests/releases/download/${EEST_VERSION}/fixtures_develop.tar.gz \
  --sim.timelimit 60m
```

Note: The `disable_strict_exception_matching` flag is only added when Phase 0 detects
that the EEST `ErigonExceptionMapper` at the discovered tag is missing required entries
(e.g. `BlockException.GAS_USED_OVERFLOW`, BAL exception types). Once upstream updates
their tagged releases with the mapper fixes (already on `forks/amsterdam` HEAD), this
flag will no longer be needed automatically.

### Phase 3: Parse Results

After each suite, parse the output to extract results:
```bash
status_line=$(tail -2 output.log | head -1 | sed -r "s/\x1B\[[0-9;]*[a-zA-Z]//g")
suites=$(echo "$status_line" | sed -n 's/.*suites=\([0-9]*\).*/\1/p')
tests=$(echo "$status_line" | sed -n 's/.*tests=\([0-9]*\).*/\1/p')
failed=$(echo "$status_line" | sed -n 's/.*failed=\([0-9]*\).*/\1/p')
```

Also check the JSON result files in `workspace/logs/*.json` for detailed per-test
results.  Report pass/fail counts and list any failing test names.

### Phase 4: Cleanup

**Always run cleanup**, even if tests fail:

```bash
# Clean up hive containers
./hive --cleanup

# Optionally remove the work directory
rm -rf "$WORKDIR"

# Prune dangling docker images from the test run
docker image prune -f
```

## Troubleshooting

### Timeout failures
Run suites separately instead of combining them. Increase `--sim.timelimit` if needed.

### Leftover containers
Run `./hive --cleanup` or `docker rm -f $(docker ps -aq)` to remove stale containers.
The hive binary has built-in cleanup: `./hive --cleanup --cleanup.older-than 1h`.

---

Source: https://www.openparable.com/skill/testing-quality/hive-test (OpenParable). Cite the canonical URL above.
