OpenParable
Skip to content
← Back to skills

CI Reproduce On GCP Vm

Testing & Quality
ci-reproduce-on-gcp-vm

CI Reproduce On GCP Vm is an Agent Skill for testing & quality published in the projectcalico/calico repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Reproduce CI test failures on a GCP VM matching the CI environment. Use when a CI job fails and the issue cannot be reproduced locally (e.g., kernel-dependent BPF verifier failures, kernel version-specific bugs).

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/projectcalico/calico/blob/HEAD/.claude/skills/ci-reproduce-on-gcp-vm/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/ci-reproduce-on-gcp-vm, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Overview

CI runs Felix tests on GCP VMs using specific Ubuntu image families. The local dev machine may have a different kernel, so some failures only reproduce on the CI kernel. This skill creates a GCP VM matching the CI environment, runs the failing test, and cleans up.

Prerequisites

  • gcloud CLI authenticated with access to the tigera-dev project
  • The calico repo checked out locally with the failing branch

Step 1: Identify the CI Image Family

Check felix/.semaphore/fv-prologue to find the image family for the failing CI job. The mapping is based on FELIX_TEST_GROUP:

Test group patternIMAGE_FAMILYUbuntu version
22.04ubuntu-2204-lts22.04 Jammy
24.04ubuntu-2404-lts-amd6424.04 Noble
25.10ubuntu-2510-amd6425.10 Plucky

For example, the bpf-24.04-ipt-with-ut test group uses ubuntu-2404-lts-amd64.

If in doubt, read felix/.semaphore/fv-prologue and .semaphore/vms/vm-bootstrap.sh for the latest mappings.

Step 2: Create the VM

zone=us-central1-a
vm_name=<user>-debug
image_family=ubuntu-2404-lts-amd64  # from Step 1

gcloud config set project tigera-dev
gcloud --quiet compute instances create "${vm_name}" \
  --zone=${zone} \
  --image-family=${image_family} \
  --image-project=ubuntu-os-cloud \
  --machine-type=n4-highcpu-4 \
  --boot-disk-size=20G \
  --boot-disk-type=hyperdisk-balanced

The machine type and disk size above match CI defaults (see felix/.semaphore/fv-prologue).

Step 3: Wait for SSH and Install Dependencies

The VM bootstrap in CI is done by .semaphore/vms/vm-bootstrap.sh. Replicate its key steps:

ssh_cmd="gcloud --quiet compute ssh --zone=${zone} ubuntu@${vm_name} --"

# Wait for SSH
for i in $(seq 1 10); do
  ${ssh_cmd} echo "SSH ready" && break
  sleep 2
done

# Install prerequisites
${ssh_cmd} "sudo apt-get update -y && sudo apt-get install -y --no-install-recommends apt-transport-https ca-certificates curl software-properties-common"

# Add Docker repo (DEB822 format, matching .semaphore/vms/vm-bootstrap.sh)
${ssh_cmd} "sudo install -d -m 0755 /etc/apt/keyrings && curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /tmp/docker.gpg && sudo mv /tmp/docker.gpg /etc/apt/keyrings/docker.asc && sudo chmod 0644 /etc/apt/keyrings/docker.asc"
${ssh_cmd} "ubuntu_codename=\$(. /etc/os-release && echo \"\${UBUNTU_CODENAME:-\$VERSION_CODENAME}\") && printf '%s\n' 'Types: deb' 'URIs: https://download.docker.com/linux/ubuntu' \"Suites: \${ubuntu_codename}\" 'Components: stable' 'Architectures: amd64' 'Signed-By: /etc/apt/keyrings/docker.asc' | sudo tee /etc/apt/sources.list.d/docker.sources > /dev/null"
${ssh_cmd} "sudo apt-get update -y"

# Install Docker and tools — pin versions to match CI (see .semaphore/vms/vm-bootstrap.sh)
# Noble (24.04): docker-ce=5:27.5.1-1~ubuntu.24.04~noble
# Jammy (22.04): docker-ce=5:20.10.14~3-0~ubuntu-jammy
# If unsure, omit the version pin to get the latest.
${ssh_cmd} "sudo apt-get install -y --no-install-recommends docker-ce docker-ce-cli docker-buildx-plugin containerd.io git make iproute2 wireguard"

# Post-install setup
${ssh_cmd} "sudo usermod -a -G docker ubuntu"
${ssh_cmd} "sudo modprobe ipip"

# Configure Docker with IPv6 (required by many FV tests)
${ssh_cmd} 'echo "{\"ipv6\": true, \"fixed-cidr-v6\": \"2001:db8:1::/64\"}" | sudo tee /etc/docker/daemon.json'
${ssh_cmd} "sudo systemctl restart docker"

# Match CI's sysctl setting (loose reverse path filtering)
${ssh_cmd} "sudo sysctl -w net.ipv4.conf.all.rp_filter=2"

Step 4: Clone Repo and Checkout Branch

# Get the current branch name
branch=$(git rev-parse --abbrev-ref HEAD)
remote_url=$(git remote get-url origin)

${ssh_cmd} "git clone ${remote_url} calico && cd calico && git checkout ${branch}"

If the remote is an SSH URL and the VM doesn't have SSH keys, use the HTTPS URL instead:

# Convert [email protected]:user/repo.git to https://github.com/user/repo.git
https_url=$(echo "${remote_url}" | sed 's|[email protected]:|https://github.com/|')
${ssh_cmd} "git clone ${https_url} calico && cd calico && git checkout ${branch}"

Step 5: Run the Failing Test

Check the kernel version first to confirm it differs from local:

${ssh_cmd} "uname -r"

Then run the specific test. Common patterns:

# BPF unit test (e.g., verifier loadability)
${ssh_cmd} "cd calico/felix && make FOCUS=TestPrecompiledBinariesAreLoadable ut-bpf"

# Specific BPF unit test
${ssh_cmd} "cd calico/felix && make FOCUS=TestNATNodePortNoFWD ut-bpf"

# Felix FV test
${ssh_cmd} "cd calico/felix && make fv GINKGO_FOCUS='TestName'"

# BPF FV test
${ssh_cmd} "cd calico/felix && make fv-bpf GINKGO_FOCUS='TestName'"

The first run will be slow (pulls Docker build images). Subsequent runs are faster.

Step 6: Clean Up

Always delete the VM when done:

gcloud --quiet compute instances delete ${vm_name} --zone=${zone}

Reference: CI Configuration Files

FilePurpose
felix/.semaphore/fv-prologueMaps test groups to image families, sets env vars
.semaphore/vms/vm-bootstrap.shVM startup script (Docker install, sysctl, IPv6)
.semaphore/vms/run-tests-on-vmsOrchestrates VM creation and test execution
.semaphore/vms/configure-test-vmPer-VM configuration after bootstrap
.semaphore/semaphore.yml.d/blocks/20-felix.ymlFelix CI job definitions and test groups