OpenParable
Skip to content
← Back to skills

Running Benchmarks

Testing & Quality
running-benchmarks

Running Benchmarks is an Agent Skill for testing & quality published in the elastic/rally repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Run Rally benchmarks (races) against Elasticsearch — an existing/external cluster or a Rally-provisioned distribution — and read the summary report. Use when running a race (any pipeline, track, challenge, target-hosts, or auth) or when interpreting throughput, latency, and service_time results.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/elastic/rally/blob/HEAD/skills/running-benchmarks/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/running-benchmarks, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Running Rally benchmarks

Launch a race against Elasticsearch and interpret its summary report. Rally can run benchmarks several ways (see Choose a pipeline).

For meaningful results, prefer an existing/external cluster you control: running Rally locally co-locates the load driver and the system under test on one host, where they can perturb each other's measurements. Local deployment is a good fit when that's not a concern — smoke testing, fast iterative local development, or demos.

Safety

  • Never run Rally as root — Elasticsearch refuses to start with root privileges.
  • Strongly avoid benchmarking production. Rally mutates cluster state (creating, writing to, and deleting indices), and competing traffic can perturb results. Prefer a dedicated, quiet environment.

Choose a pipeline

Rally infers the pipeline from your flags.

PipelineUse whenSelected by
benchmark-onlyPrimary path. Cluster is already running and you provisioned it.--pipeline=benchmark-only --target-hosts=...
from-distributionRally downloads and runs Elasticsearch locally. Setup checks/demos; sharing a host skews results, so only large differences stand out.--distribution-version=X
from-sourcesRally builds Elasticsearch from a git revision and runs it locally (CI/dev). Setup checks/demos; sharing a host skews results, so only large differences stand out.--revision=...

benchmark-only trade-off (docs/pipelines.rst): because Rally did not provision the cluster, results are not easily reproducible and Rally cannot gather host-level metrics (CPU, GC, disk I/O, index size). Treat those numbers as directional.

Launch a race

  1. Discover tracks, then inspect one:

    esrally list tracks
    esrally info --track=pmc
    
  2. Run against the cluster:

    esrally race --track=pmc --pipeline=benchmark-only \
      --target-hosts=10.5.5.10:9200,10.5.5.11:9200
    
  3. Secured cluster — TLS + basic auth via --client-options:

    esrally race --track=pmc --pipeline=benchmark-only --target-hosts=host:9243 \
      --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'changeme'"
    

    API-key auth (including Elastic Cloud / Serverless — see docs/serverless.rst):

    esrally race --track=geonames --pipeline=benchmark-only --target-hosts=${ES_HOST}:443 \
      --client-options="use_ssl:true,api_key:${ES_API_KEY}" --on-error=abort
    
  4. Record the race-id so you can find the results later. Rally assigns a random UUID by default; pass your own with --race-id to know it up front:

    RACE_ID=$(uuidgen)
    esrally race --track=pmc --pipeline=benchmark-only --target-hosts=host:9200 --race-id="$RACE_ID"
    echo "$RACE_ID"   # note this for `esrally compare` / metrics-store queries
    

    If Rally generated the id, recover the most recent one afterwards with esrally list races. (Optionally add --user-tags="key:value" for human-friendly filtering — not required.)

  5. Save the report: --report-file=~/benchmarks/result.md (add --report-format=csv for CSV).

  6. Fast setup/smoke check: add --test-mode to ingest a tiny slice instead of the full corpus.

Multi-cluster A/B (benchmark-only only; telemetry disabled in this mode): add --multi-cluster with JSON-format --target-hosts/--client-options. See docs/recipes.rst.

Races are long-running (often 20+ minutes) and download large corpora on the first run for a track. Treat a race as a long job, not a quick command.

Validate correctness before trusting numbers

Rally does not abort on query errors by default — it folds them into the error rate in the summary, which silently skews results.

  • Confirm operations return the hits you expect. A mapping mismatch (e.g. text vs keyword) yields 0 hits with no error.
  • While validating, use --on-error=abort and/or add track-level assertions and run with --enable-assertions.
  • A non-zero error rate on a task means those results are suspect. See docs/recipes.rst.

Read the summary report

  • throughput — operations per second (higher is better).
  • latency — includes the time a request waits in the queue before Rally sends it.
  • service_time — request→response only, excludes wait (what most load-test tools incorrectly call "latency").
  • processing_time — includes Rally's client-side overhead; a large gap vs service_time points to a client-side bottleneck.
  • Metrics are reported per task; "Cumulative … of primary shards" is not wall-clock time.
  • Check error rate first as a validity gate.

See docs/summary_report.rst and docs/metrics.rst.

References

  • Running & pipelines: docs/race.rst, docs/pipelines.rst
  • Recipes (existing cluster, Elastic Cloud, multi-cluster): docs/recipes.rst
  • Serverless: docs/serverless.rst
  • Report & metric definitions: docs/summary_report.rst, docs/metrics.rst
  • All flags (do not invent flags; verify here or via esrally <subcommand> --help): docs/command_line_reference.rst
  • Comparing and charting results across past races: the accessing-benchmark-results skill