ci-runner-utilization

CI Runner Utilization & Downsizing Analysis

Analyzes CPU and memory utilization of self-hosted CI runners in the camunda/camunda repository to find overprovisioned jobs and recommend cheaper runner types.

What costs money and what doesn't

Self-hosted runners cost money — these are Kubernetes pods on GCP or AWS billed by core-hour. Their runner_type starts with gcp- (e.g., gcp-perf-core-16-default) or aws-. More cores = higher cost. Downsizing from 16 to 8 cores roughly halves the per-job compute cost.

GitHub-hosted runners are free for public repos — jobs on ubuntu-latest / ubuntu-slim have runner_type = NULL in BigQuery. Ignore them entirely for cost optimization.

CPU is the expensive resource. Memory is proportional to cores and much cheaper per unit. Focus downsizing decisions on CPU utilization; only check memory to ensure a smaller runner won't OOM.

Perf runners cost more than standard runners — gcp-perf-core-N uses faster CPUs than gcp-core-N. Only suggest downgrading perf→standard if the job doesn't need fast CPUs (e.g., linting, static analysis, artifact uploads).

Longrunning runners cost more than default — -longrunning has higher durability guarantees and costs more. Only needed for jobs that genuinely run long or are release-critical.

Runner type naming convention

Format: {cloud}-{tier?}-core-{cores}-{durability}

Available self-hosted runner types can be found on https://github.com/camunda/infra-global-github-actions/blob/main/actionlint/actionlint.yaml

Downsizing follows the same family: gcp-perf-core-16-default → gcp-perf-core-8-default.

Prerequisites

• bq CLI authenticated with access to project ci-30-162810
  • Verify: bq query --use_legacy_sql=false 'SELECT 1'
• Data is in ci-30-162810.prod_ci_analytics.build_status_v2 (90-day retention)
• CPU/memory metrics were added on 2026-05-18 — data availability starts from that date

How to analyze

Step 1: Identify underutilized self-hosted jobs

This query finds jobs where the peak CPU p95 never exceeds 50% of the runner's capacity, grouped by runner type. Only self-hosted runners (non-NULL runner_type) are included.

bq query --use_legacy_sql=false --format=prettyjson '
SELECT
  job_name,
  runner_type,
  COUNT(*) AS samples,
  ROUND(AVG(cpu_usage_ratio_p95), 3) AS avg_cpu_p95,
  ROUND(MAX(cpu_usage_ratio_p95), 3) AS max_cpu_p95,
  ROUND(AVG(memory_usage_ratio_p95), 3) AS avg_mem_p95,
  ROUND(MAX(memory_usage_ratio_p95), 3) AS max_mem_p95
FROM `ci-30-162810.prod_ci_analytics.build_status_v2`
WHERE cpu_usage_ratio_p95 IS NOT NULL
  AND ci_url LIKE "%camunda/camunda%"
  AND runner_type IS NOT NULL
  AND (runner_type LIKE "gcp-%" OR runner_type LIKE "aws-%")
GROUP BY job_name, runner_type
HAVING MAX(cpu_usage_ratio_p95) <= 0.5
ORDER BY max_cpu_p95 ASC
'

Step 2: Get full utilization picture (all self-hosted jobs)

This shows all jobs sorted by CPU usage so you can see the full spectrum and identify the boundary between "needs downsizing" and "correctly sized":

bq query --use_legacy_sql=false --format=csv --max_rows=200 '
SELECT
  job_name,
  runner_type,
  COUNT(*) AS samples,
  ROUND(AVG(cpu_usage_ratio_p95), 3) AS avg_cpu_p95,
  ROUND(MAX(cpu_usage_ratio_p95), 3) AS max_cpu_p95,
  ROUND(AVG(memory_usage_ratio_p95), 3) AS avg_mem_p95,
  ROUND(MAX(memory_usage_ratio_p95), 3) AS max_mem_p95
FROM `ci-30-162810.prod_ci_analytics.build_status_v2`
WHERE cpu_usage_ratio_p95 IS NOT NULL
  AND ci_url LIKE "%camunda/camunda%"
  AND runner_type IS NOT NULL
  AND (runner_type LIKE "gcp-%" OR runner_type LIKE "aws-%")
GROUP BY job_name, runner_type
ORDER BY max_cpu_p95 ASC
'

Step 3: Check runner type distribution

Understand which runner types carry the most jobs and runs:

bq query --use_legacy_sql=false --format=prettyjson '
SELECT
  runner_type,
  COUNT(DISTINCT job_name) AS distinct_jobs,
  COUNT(*) AS total_runs,
  ROUND(AVG(cpu_usage_ratio_p95), 3) AS overall_avg_cpu_p95
FROM `ci-30-162810.prod_ci_analytics.build_status_v2`
WHERE ci_url LIKE "%camunda/camunda%"
  AND cpu_usage_ratio_p95 IS NOT NULL
  AND runner_type IS NOT NULL
GROUP BY runner_type
ORDER BY total_runs DESC
'

Step 4: Deep-dive a specific job (time series)

When you want to see if a job's usage is stable or has spikes over time:

bq query --use_legacy_sql=false --format=prettyjson '
SELECT
  report_time,
  job_name,
  runner_type,
  ROUND(cpu_usage_ratio_p95, 3) AS cpu_p95,
  ROUND(memory_usage_ratio_p95, 3) AS mem_p95,
  build_status
FROM `ci-30-162810.prod_ci_analytics.build_status_v2`
WHERE ci_url LIKE "%camunda/camunda%"
  AND job_name = "REPLACE_WITH_JOB_NAME"
  AND cpu_usage_ratio_p95 IS NOT NULL
ORDER BY report_time DESC
LIMIT 50
'

How to interpret results and make recommendations

Utilization metrics

• cpu_usage_ratio_p95: 95th-percentile CPU usage as a fraction of the container's CPU limit (0.0–1.0). A value of 0.25 on a 16-core runner means the job used ~4 cores at p95.
• memory_usage_ratio_p95: Same for memory. Check this to ensure a smaller runner won't OOM.
• Always use MAX(cpu_usage_ratio_p95) across runs, not just the average — you need to handle the worst case, not the typical case.

Decision framework

Memory safety check

Before recommending a downsize, verify max_mem_p95:

• If max_mem_p95 < 0.5 on the current runner, halving cores (and thus memory) is safe.
• If max_mem_p95 > 0.5, halving would risk OOM. Consider keeping the larger runner or only stepping down one size (16→8 instead of 16→4).

If no suitable runner type can be found, suggest creating new runner types.

Sample count matters

• ≥ 100 samples: High confidence — safe to act on.
• 30–100 samples: Medium confidence — recommend with a note to monitor.
• < 30 samples: Low confidence — flag for future review, don't act yet.

Forming the recommendation

For each underutilized job:

1. Note the current runner_type and extract the core count.
2. Multiply max_cpu_p95 by the core count to get effective cores used.
3. Find the smallest available runner type (same family) that provides ≥1.5x the effective cores.
4. Check memory won't OOM on the smaller runner.
5. State: job name, current runner, suggested runner, CPU headroom, memory headroom, sample count.

Example: A job with max_cpu_p95 = 0.25 on gcp-perf-core-8-default uses ~2 effective cores. A gcp-perf-core-4-default (4 cores) gives 2x headroom → recommend it.

If no suitable runner type can be found, suggest creating new runner types.

Implementing the recommendation

For each underutilized job:

1. Ask the user for confirmation to apply the recommendation.
2. Find the GitHub Action workflow YAML file that contains the job, and adjust the runs-on: label.
3. Offer to commit and push the changes to a Pull Request, and observe the CI runtime behavior on that PR.
4. Confirm job run times on the PR do not increase meaningfully.

BigQuery table schema reference

Table: ci-30-162810.prod_ci_analytics.build_status_v2 (90-day retention)

Data collection pipeline

1. start-build-monitor action starts a background monitor (5s polling) collecting CPU/memory from cgroups v2/v1 or /proc/
2. submit-build-status action stops the monitor, aggregates stats (avg, p95) via AWK, reads runner_type from /home/runner/.camunda-arc-runner-info/runs-on, and POSTs to BigQuery
3. Metrics are normalized ratios relative to the container's CPU/memory limits

Source: camunda/infra-global-github-actions/start-build-monitor/ and camunda/infra-global-github-actions/submit-build-status/