Nic CI Pipelines

DevOps & Security

nic-ci-pipelines

Nic CI Pipelines is an Agent Skill for devops & security published in the nginx/kubernetes-ingress repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

CI/CD pipeline structure, GitHub Actions workflows, reusable workflow patterns, and matrix builds for NIC. Use when working on CI workflows, debugging build failures, adding new workflow steps, modifying build matrices, or understanding the release pipeline.

View on GitHub ↗

QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

Open your project in Codex.
Copy and paste the prompt below.
Review files and risks before approving any action.

Prompt to paste

Inspect the Agent Skill at https://github.com/nginx/kubernetes-ingress/blob/HEAD/.github/skills/nic-ci-pipelines/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/nic-ci-pipelines, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

NIC CI/CD Pipelines

Workflow Architecture

The CI system uses GitHub Actions with extensive reusable workflow composition.

ci.yml (main CI orchestrator)
  -> checks (format, lint, codegen, CRDs, chart version)
  -> unit-tests
  -> build-artifacts.yml (reusable)
       -> build-oss.yml (per-variant, matrix)
       -> build-plus.yml (per-variant, matrix)  <- also used for NAP variants
  -> helm-tests
  -> setup-smoke.yml (reusable)
  -> e2e tests

image-promotion.yml (post-merge)
  -> builds images, tags edge/stable
  -> Trivy + DockerScout security scans
  -> publishes edge Helm charts

release.yml (manual dispatch)
  -> oss-release.yml
  -> plus-release.yml
  -> publish-helm.yml
  -> certify-ubi-image.yml
  -> marketplace pushes (AWS, Azure, GCP)

Key Workflows

Workflow	Trigger	Purpose
`ci.yml`	PR to `main`/`release-*`, merge_group, workflow_dispatch	Main CI: checks + build + test
`lint-format.yml`	PR to `main`, merge_group	goimports, gofumpt, golangci-lint, actionlint
`regression.yml`	Daily cron (03:00 UTC), manual	Multi-K8s-version regression
`image-promotion.yml`	Push to `main`/`release-*`	Post-merge image tagging + scanning
`release.yml`	Manual dispatch	Full release orchestrator
`build-base-images.yml`	Weekday cron (04:30 UTC)	Rebuilds all base images

Release Sub-Workflows (called by `release.yml`)

Workflow	Purpose
`oss-release.yml`	OSS image release
`plus-release.yml`	Plus/NAP image release
`publish-helm.yml`	Helm chart publishing to registry

Reusable Build Workflows (called via `workflow_call`)

Workflow	Purpose
`build-artifacts.yml`	Orchestrates GoReleaser binary builds + image matrix
`build-oss.yml`	Builds single OSS image variant
`build-plus.yml`	Builds single Plus/NAP image variant
`build-test-image.yml`	Builds Python e2e test image
`setup-smoke.yml`	Sets up and runs smoke tests
`patch-image.yml`	OS-level patches on existing images
`retag-images.yml`	Re-tags images in GCR Dev Registry

Security & Compliance

Workflow	Purpose
`codeql-analysis.yml`	GitHub CodeQL scanning
`scorecards.yml`	OpenSSF Scorecards
`dependency-review.yml`	Dependency review for PRs
`mend.yml`	Mend (WhiteSource) software composition analysis
`certify-ubi-image.yml`	Red Hat UBI certification for OpenShift

CI Patterns

Matrix Builds

Image variants are defined in JSON under .github/data/:

matrix-images-oss.json: debian, alpine, ubi (amd64 + arm64)
matrix-images-plus.json: debian-plus, alpine-plus, alpine-plus-fips, ubi-10-plus
matrix-images-nap.json: WAF v4/v5, DoS, UBI 10 (amd64 only)
matrix-smoke-oss.json, matrix-smoke-plus.json, matrix-smoke-nap.json: Smoke test matrices
matrix-regression.json: Regression test matrix (K8s version combinations)
patch-images.json: Patch image definitions for patch-image.yml

Caching Strategy

Go binaries: cached by go_code_md5 hash
Docker images: cached by docker_md5 hash
Stable images in GCR are checked before rebuilding

Fork Awareness

forked_workflow variable gates authenticated operations. Forked PRs get local-only builds without secret access.

Concurrency

Each workflow uses group: ${{ github.ref_name }}-<suffix> with cancel-in-progress: true.

Secrets

Retrieved from Azure Key Vault via OIDC -- not stored as GitHub secrets directly.

Version Source of Truth

.github/data/version.txt contains IC_VERSION and HELM_CHART_VERSION.

Gotchas

Never add secrets as GitHub repository secrets -- use Azure Key Vault OIDC flow
Always pin GitHub Actions to immutable SHA hashes, not mutable tags
Matrix JSON files in .github/data/ must stay in sync with Makefile image targets
NAP variants are linux/amd64 only -- do not add arm64 to NAP matrices
Renovate manages tool versions via # renovate: comments -- do not update manually
image-promotion.yml runs on merge to main, not on PR -- don't expect images from PRs