vllm-bench

vllm-bench

vllm-bench is a Rust benchmark client for online LLM serving. It does not run the model. It drives an already running OpenAI-compatible or vLLM endpoint and reports TTFT, TPOT, ITL, E2EL, throughput, and multi-turn per-turn breakdowns.

Use this workflow for multi-turn work:

1. Install or locate vllm-bench.
2. Run vllm-bench --help and use it as the authority for current flags.
3. Confirm the server base URL and model id from /v1/models.
4. Dry-run the dataset locally.
5. Run a tiny smoke benchmark.
6. Scale conversations, turns, and concurrency only after the smoke passes.
7. Save JSON for any result that may be compared later.

Install

Install from the upstream repository:

cargo install --git https://github.com/vllm-project/vllm-bench vllm-bench

The trailing vllm-bench package name matters: the repo also contains another binary, so omitting it can fail with "multiple packages with binaries found". Cargo installs the binary to ~/.cargo/bin/vllm-bench.

Alternatives from the README:

curl -fsSL https://github.com/vllm-project/vllm-bench/releases/latest/download/vllm-bench-$(uname -m)-linux-musl \
  -o vllm-bench && chmod +x vllm-bench

git clone https://github.com/vllm-project/vllm-bench.git
cd vllm-bench
./install.sh

After install, run:

vllm-bench --version
vllm-bench --help

Choose Backend

For multi-turn chat, always use:

--backend openai-chat

Common backends:

Prefer http://127.0.0.1:<port> over localhost; some servers bind IPv4 while localhost resolves to IPv6.

Confirm the model id before benchmarking:

curl -s http://127.0.0.1:8000/v1/models

Use the returned id as --model, or omit --model and let vllm-bench auto-fetch it.

Multi-Turn Flags

Core flags:

--multi-turn
--multi-turn-num-turns <N>
--num-prompts <CONVERSATIONS>
--multi-turn-concurrency <CONCURRENT_CONVERSATIONS>

In multi-turn mode, --num-prompts means conversation count, not request count. A run with --num-prompts 50 --multi-turn-num-turns 5 can send up to 250 chat requests.

Synthetic random conversations:

--dataset-name random
--random-input-len <TOKENS_FOR_TURN_1>
--per-turn-input-len <TOKENS_FOR_TURNS_2_PLUS>
--random-output-len <OUTPUT_TOKENS_PER_TURN>

If --per-turn-input-len is omitted or 0, later turns reuse --random-input-len. Use --multi-turn-min-turns and --multi-turn-max-turns when conversation lengths should vary.

Think time and prefix sharing:

--multi-turn-delay-ms 500
--multi-turn-prefix-global-ratio 0.2
--multi-turn-prefix-conversation-ratio 0.5

Prefix sharing works only with --dataset-name random; the two ratios must sum to less than 1.0.

When either prefix-sharing flag is non-zero, vllm-bench switches to fixed- length per-turn prompts with no accumulated chat history. Use this only for prefix-cache experiments; leave both ratios at 0 for realistic multi-turn conversation growth.

Dry Run

Dry-run before sending traffic. It loads the tokenizer and builds conversations without hitting the server:

vllm-bench \
  --backend openai-chat \
  --model Qwen/Qwen3-4B \
  --tokenizer <model-path> \
  --dataset-name random \
  --multi-turn --multi-turn-num-turns 2 \
  --random-input-len 16 --random-output-len 4 \
  --num-prompts 2 \
  --dry-run

<model-path> is a local tokenizer/model directory; for a public example, replace it with the served model id or any local model directory.

Example: Generic vLLM Multi-Turn

Use this after a vLLM OpenAI server is already running on port 8000:

vllm-bench \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --model <model-id-from-/v1/models> \
  --dataset-name random \
  --multi-turn \
  --multi-turn-num-turns 5 \
  --random-input-len 512 \
  --per-turn-input-len 256 \
  --random-output-len 128 \
  --num-prompts 50 \
  --multi-turn-concurrency 10 \
  --ready-check-timeout-sec 60 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,99 \
  --save-result

Read the output as:

• Conversations completed/total: end-to-end conversation success.
• Successful requests: successful turns.
• Total input tokens: includes growing chat history in normal multi-turn mode.
• per_turn_metrics in JSON: TTFT/TPOT/ITL/E2EL by turn index.

Expect later turns to have larger input token counts because each request sends the prior user/assistant history plus the next user message.

Example: Local OpenInfer Qwen3-4B Smoke

This was validated locally with OpenInfer Qwen3-4B and vllm-bench built from this repo. Start OpenInfer:

cd <openinfer-repo>
cargo run --release -p openinfer-server -- \
  --model-path <model-path> \
  --served-model-name Qwen/Qwen3-4B \
  --port 18080 \
  --cuda-graph=false

Confirm the model:

curl -s http://127.0.0.1:18080/v1/models

Run a tiny multi-turn benchmark:

cd <vllm-bench-repo>
target/release/vllm-bench \
  --backend openai-chat \
  --base-url http://127.0.0.1:18080 \
  --model Qwen/Qwen3-4B \
  --tokenizer <model-path> \
  --dataset-name random \
  --multi-turn \
  --multi-turn-min-turns 2 \
  --multi-turn-max-turns 3 \
  --random-input-len 32 \
  --per-turn-input-len 12 \
  --random-output-len 8 \
  --num-prompts 3 \
  --multi-turn-concurrency 2 \
  --ready-check-timeout-sec 30 \
  --extra-body '{"min_tokens":null}' \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90 \
  --save-result \
  --result-dir /tmp \
  --result-filename vllm-bench-multi-turn-smoke.json

The --extra-body '{"min_tokens":null}' is for OpenInfer compatibility. The random multi-turn path auto-adds min_tokens unless overridden; this pins output length on vLLM but OpenInfer currently rejects that field. --ignore-eos also skips min_tokens and worked in local smoke testing, but it asks the server to ignore EOS and can grow contexts more aggressively.

The local smoke run completed 3/3 conversations, 8/8 turns, and wrote /tmp/vllm-bench-multi-turn-smoke.json with per_turn_metrics.

Scale Safely

After the smoke passes:

1. Increase --num-prompts first to improve measurement stability.
2. Increase --multi-turn-concurrency to find the service knee.
3. Increase --random-input-len, --per-turn-input-len, and --random-output-len to match the target workload.
4. Save every serious run with --save-result --metadata KEY=VALUE.

For deterministic latency A/B runs, pass --temperature 0 explicitly. Omit it only when the benchmark is intentionally measuring server default generation behavior.

Use a sweep when searching for the concurrency knee:

vllm-bench \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --model <model-id> \
  --dataset-name random \
  --multi-turn --multi-turn-num-turns 5 \
  --random-input-len 512 --per-turn-input-len 256 --random-output-len 128 \
  --num-prompts 200 \
  --sweep-max-concurrency 1,2,4,8,16,32 \
  --sweep-summary-percentiles 90,99 \
  --save-result

For A/B comparisons, keep the workload flags identical and compare result JSON:

vllm-bench --compare baseline.json candidate.json

Troubleshooting

• min_tokens is not supported by this engine: pass --extra-body '{"min_tokens":null}' or, if appropriate, --ignore-eos.
• All-zero metrics with failed requests: inspect the first HTTP error printed above the result block; it usually names the unsupported request field.
• Connection fails on localhost: switch to 127.0.0.1.
• Model id mismatch: use curl /v1/models and set --model to that exact id.
• Tokenizer load fails: set --tokenizer to a local model directory with tokenizer.json, or rely on server-side tokenize/detokenize fallback if the server implements it.
• Dataset construction is suspicious: run the same command with --dry-run.
• Long serious runs need reproducibility: set --seed, --result-filename, and --metadata entries for server commit, model, GPU, and key flags.

Metrics

Use database-style thinking: requests are transactions, prompt tokens are read set size, output tokens are write volume, and concurrency is the number of in-flight conversations.

• TTFT: queueing plus prefill latency until the first output token.
• TPOT: decode time per output token after the first token.
• ITL: per-token latency distribution.
• E2EL: total turn latency.
• Request throughput: turns per second.
• Output throughput: generated tokens per second.
• Multi-turn JSON: inspect per_turn_metrics to see how latency changes as history grows.